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FOREWORD 



The Fort «*Knox Field Unit has been highly successful in applying psycho- 
logical principles and educational technology in solving Army trailing 
problems. The training evaluation/ feedback team of this unit performs 
research and develop ment aimed at providing the Army with practical methods 
for assessing and improving the effectiveness of its training programs arid 
materials. . 

In the past the Army has not had an effective method for evaluating and 
improving their training programs. Previous attempts to evaluate training 
have either relied heavily on the perceptions 6f the trainers and the 
soldiers being trained, or have depended on the use of harris-on testing 
as the_ sole s&irde of evaluation data. Neither of these methods has been 
entirely satisfactory. Perceptions of the trairtbig by trainers or trainees 
.seldom ^provide the kirri£ of information necessary to identify a«3 correct 
specific training problems. And while hands-on testing may identify 
the tasks for which training is deficient/ hands-on tests usually do not 
provide any information about the probable cause of poor task performance. 
Without information about probable causes > specific action cannot be taken 
to eliminate the training deficiencies. 

Over a three year period, the training evaluation/ feedback team has devel- 
oped a unique system fqr . evaluating training programs that- enables • the 
evaluator to identify and correct the npgt probable causes of poor task 
performance. Detailed guidance in using this, new system i$ provided 
in a set of easy-to-use jobjaids designed specifically for evaluating 
•Army training programs. The Training Program Evaluation (TPE) methodology 
described in these job aids, vas developed by progress ively refining proto- 
type data collection procedures ar£ formats through a series of field trials 
- in vrtiich TPE was tested with 9 typical users against typical Army training 
This report traces the development and field trial of the TPE systen from 
its inception to the completed. TPE job aids. The , information presided in 
this report on the fie^d" trial pf the TPE system arid its associated job aids 
may prove valuable to anyone tasked with ccrkducting a large scale training . 
program evaluation, particularly against a new system during the operational, 
testing phases of the Life Cycle Systems Management Model . 



■ : ■ ■ PREFACF. 

%■ 

. ' This report traces the development arid"- rield trial of a system for evalu- 
ating training programs; Although the report includes considerable information 
on the evaluation system itself, no attempt has been made to fatty. describe 
the system or to demonstrate how it is used'. That information is amply docu- 
mented elsewhere -(Witmer, Note 5; Kristiansen, Note 6; Kristiansen and Wither, 
Note 7; and Kristiansen and Witmer, Note 8). Similarly, though much ^is said 
'abotft the Ml Ot-iii in connection with the field trial of the training evalua- 
tion system, this report makes no attempt to provide a complete account of all 
the events associated with the Ml OT-III. Background information on the HI 
OT-III is presented only insofar as it relates to the development. and field 
trial of the Training Program Evaluation (TPE) job aids and, procedures. 

The purpose of this report is twofold: (1) To trace the development of 
the TPE system from its inception, through the Ml OT-III field trial, .to the 
developed system as described in the revised TPE job aids, and (2) To explain 
how ARi's experiences in using the TPE system, particularly during the Ml 
OT-III, led to changes in the TPE system. v, 
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RKQHiREMKNT ■ . 

/ . Vf 

in April 197,8, the Army Tra/iling Study Croup 7 (ARTS) asked ARI tcv develop 
job aids and methodological guides for assessing the effectiveness and effi- 
ciency of Army training programs. ARTS further requested that the Ml tank 
system he used as a tost bed to test the adequacy of any materials or method- 
ologies developed. This report documents the process by which ARI. developed 
and field tested the requested job aids and methodologies. 

PROCEDURE " ' 

• * . . ■ r 

A system for assessing the effectiveness and efficiency of training pro- 
grams, henceforth referred to as Training Program Evaluation (TPE) 3 was de- 
veloped by alternately designing, trying out, and revisin^data collection 
formats and procedures; ARl's initial response to the ARTS request , which was 
to. develop a 'Training Observation Form a^d a Training Opinion Questionnaire 
for evaluating training during the Ml OT-II indicated, the need for additional • 
development of training evaluation materials. TPE used as its starting point 
the procedures anc! data collection formats developed for ARI by Hat less Per- 
formance Cuild, Inc. Jhese procedures and formats, known as the llarlcss Guide- 
lines, were tested against, several courses routinely conducted 'at the Armor * 
Center with encouraging results. The early trial*/ 6f the Harlcss Guidelines ' 
at fc the Armor Center, while encouraging, indicated/ that additional work was re- 
quired to develop TPE to the point that it could be Jjsed routinely by Army 
personnel to effectively evaluate training. - 

Aft er further mod if ica t ions by ARI j the TPE system was ready to undergo 
the major test of its usefulness as a method for evaluating training/f^ogTams^T 
TPE was to be used to evaluate the effectiveness and efficiency of J^ie training 
designed to transition soldiers from the M60Al tank to the Ml tank during the 
Ml. OT-III.' An ART researcher was present on site during the OT to analyze the 
TPE data and assess the usefulness of TPE in evaluating the Ml transition 
training prograirf? Ba^sed on lessons learned from the OT-III experience, ARI 
further revi<rtkl the TPE materials. 

FINDINGS - ' i tat 

* The, irif dririat ion provided by the TPE system during the Ml OT-III was 
^found Useful by all of the agencies involved in the evaluation of the Ml transi- 
tion training program. The data provided by the TPE system were used to recom- 
mend changes in. the transition training program. Many of the recommended f 
changes- were adopted by the trainers, with resulting increases in soldier /pro- ' 
ficiency on the end-rof -block tests. Some of the NCO's collecting the TPE? data 
during the fie&d trial failed to make the necessary observations required \y 
the TPE system] opting instead to record their subjective^ impressions of the 
training. To Remedy this Situation, the TPE materials were revised following 
the field trial in a way that ^encourages data collectors to record only what 
they observe rather than record their opinions about the training. The revised 
TPE system is documented in a set of four user-oriented job aids (ARI Research 
ProdLict.s'*8i-15, 8i-i6; 81-17 f and Mzl&i* , — 



C 



UTILIZATION 



The iniormat ton prov tdod tn thifj report wny_bu_ii?U'f ul_ lo mil it.iry man ago r 
who nrr Riven the rrsponnibi I ity' to evaluate existing training ]}toj* rams ; The 
report traces t ho dove lopmont and f-ioid trial of piie TPH system loading to the 
finished product — the rev tsed ; TPK Job aids; It ^points out the advantages of 
TP 15 as an evaluation method, and identifies common pitfalls to avoid when ap- 
plying an evaluation system such as. ti'tf; "The lessons learned during the field 
atrial of the TPK system may help m 1 1 i ta£y managers avoid costly mistakes in 
conducting; their own training; program evaiu.it ions . 
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SECTION 1 



GENESIS OF THE TPE SYSTEM 
Irit roduct ion ' - 

The Army recognizes the need to evaluate its training programs. Due, 
however^ to the unavailability of personnel, resources, and expertise in, evalu r 
atibri methodology^ formal evaluation programs, for use in training course im- 
provement are not a standard feature of the AVmy training system. in 1976^ 
a survey of; training program evaluation activities in the military services 
indicated that ^ in the Army, individual schools did 1 i t t>l e to assess the effec- 
tiveness of the courses they conducted (Hall, Lam & Bellamy, 1976)/ In 19S1 
t|ie Comptroller General of the United States, in a report to the Congress, said, 
^e"c a~u"s~e - 1 he~~ArTny d bes no £" Tiav e ~an"~e f E e^c"Frye"Army-wi3 e - man a g ement s ysS" to " 
oversee the skill training program, . it is difficult to identify where improve- 
ments are ndeded . An effective monitoring and evaluation system is needed to 
provide Army -commanders at all" levels the program evaluation data and other 
management information needed *For informed decision making. 11 (GAO Report 
FPCD-81-29, 1981. ) t 

The training program evaluation (TPE) system discussed in this report can 
be used by Army personnel to evaluate the effectiveness and efficiency of Army 
training programs. The TPE system iden t if ies" spec if ic- training program de- 
ficiencies, allowing the program evaluator to identify the most likely causes 
of poor soldier performance. The TPE system also assists/ training managers in 
correcting training program problems by recommending courses of actions to be 
taken by the training manager in the" event that particular deficiencies are 
discovered. The TPE system is unique in that it does not rely on the percep- 
tipns of the soldiers being trained or trainers for its data. Neither does it 
require the administration of special post-tests to assess the j ef f ec t iveriess 

Of training. TPE examines the t r ^ i ni : papjvnMH i pi I 1 y ; t'hmparmg rhn TrairS^ 

~rng~~program as it is documented in the lesson plan and delivered by the in- 
structor to characteristics that according to educational technologists are 
indicative of a good training program. The TPE system requires that an ob- 
server be present on - site during the delivery of training and during the con- 
duct of the; end-of-block tests to collect information about the instructional 
procedures used in training and testing the soldiers. The information obtained 
during this observation phase is used in conjunction with information from the 
Lesson plans and soldier performance data as measured by end-of-block tests to 
identify training program deficiencies and their probable causes. By identify- 
ing causes of training program deficiencies, TPE allows the Army to eliminate 
program deficiencies by removing the causes of those deficiencies. TPE ful- 
fills the Army's need for an effective means of monitoring and evaluating train- 
ing programs. Complete information about the TPE system and how it is used is 
(locum en ted in a series of job aids (Witmcr , Note 5; Kr ist ianscn, Note 6; 
Rristiahheh and Wllmer , Note 7; and Kr ist i an sen and Witmer, Note 8). 

The eva ilia t Ion me t hod o logy Iiicor_poraLed. in . the. TPE ^.ystem-wi' duve loped 
in response to L lie i Army's need Lo evaluate the effectiveness and efficiency of 
training program/; for new weapons systems. Army planners realized Lh.it a large 
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number of new weapone systems were under development arid would be fielded in z 
the: near future. Planners such as the Army Training, Study Group- (ARTS) foresaw 
that the~ introduction of the new weapons systems would involve large-scale 
training programs designed to train soldiers to operate arid maintain the weapons, 
systems; ARTS knew that the overall effectiveness of the weapons systems was . 
linked inalterabiy to the effectiveness of the training programs developed for 
them. This was a source of concern. for ARTS because adequate ^guidelines for 
eva^ujhting Aqn? training programs did riot exist at the time. ; 

The Ml QT-I1 Training Evaluation '<.'-. 

In April 1978, the Army Research Institute (ARI) received a mid-year re- 
quest from ARTS to develop job aids and methodological guides for assessing 
the effectiveness and efficiency of Array training programs. ARTS further re- 
"q-ues ted~~t hatTt he "HIT ank" sy ^enT be_ u sia^TTest bed. " .The Ml" " wa s" a~ma jox" - 
weapon system that was already well into the Operational Testing (OT) Cycle. 
The individual training phase of the Ml OT-II was about to begin. In order to. ; 
take -advantage of the opportunity presented by the Ml OT-II, ARI quickly as- 
sembled two data collect i/5n^Tnst rumen ts to be used in evaluating the individual 
in-turret portion of th/ training package for transitioning" soldiers from the J 
M60A1 tank to the Ml tank. A Training Observation Form (Appendix A) was pre- 
pared to structure the observation, of training. In response to a request from 
the US Army Armor Center (USAARMC) , a Training Opinion Questionnaire (Appendix 
B) was developed to assess the opinions and attitudes of the OT-IT player per-" 
sonnel regarding the training during different phases of tfre OT. 



The Training Observation Form and the Training Opinion Questionnaire were 
piloted against the Ml OT-II transition training on site at Fort Bliss, Texas, 
by a team of ARI researchers. Following the OT , the data collected using 
these evaluation instruments were analyzed in an attempt to determine which of 

rh> dnffi collected wpt-p useful and which were not. ^he_Xijldings_ regarding the 

usefulness of the evaluation d-ata were shared with USAARMC and ARTS. 

The data collected during the OT-IT was not adequate* for evaluation pur- 
poses,' due in part to the unavailability of individual soldier performance data. 
• Few conclusions could be >drawn on the basis of the training observation and 
training opinion data alWeT" "Detailed analysis of the training opinion data 
indicated that it was virtually useless as a measure of training effectiveness. 
Because of the limited usefulness of the evaluation instruments developed for 
the Ml OT-II, ARI and ARTS decided %Sat an extensive developmental effort aimed 
at- producing a systematic method for conducting training program evaluations 
was heeded. ■ - . 

Contracting fox t J to: Development of TPE Materials 

Toward the end of FY 78, ARTS requested that ARI write a statement of work 
for a contract to develop TPE materials. The contract, to be let sole source 
to liar less Performance Guild, Inc., was to be funded by monies secured by ARTS 
from TRADOC , monitored jointly by the ARI Field Unit at Fort Knox and the 

USAARMC Directorate of Training Developments (DTD) and let- by the Fort Knox. 

Procurement Division of the Directorate of Industrial Operations . The state- 
ment of work was written and' the contract to develop TPE materials was let. 
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Under the terms of tti^contract , Hariess was required to develop guide- 
lines arid Job :aids_ for evaluating the effectiveness of training designed for -; 
the operational testing of the ffi. tank. Using lessons learned' during the OT-II^ 
the guides arid' job aids were t6 be designed so that they could be used by ARI 
researchers to evaluate graining dajriri^ the Ml OT-III, The guides arid job aids 
-were to include their own training materials so that formal training in their 
use would riot be necessary. Although specif icaliy designed for ARI researchers 
to use in evaluating Ml transition training , it was understood thaf the method- 
ology developed should be generalizable to any training program- associated with 
the introduction *of a new weapons system into the Army. 
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SECTION II - ; 

DEVELOPMENT AND TRIAL OF THE ORIGINAL TPE JOB AIDS • 

-T4je--Ha-i^Ljess" Guidelines - ' 

The primary product to, come out of the contract with Harless Performance 
Guild, Inc. was ARI Working Paper, FKFU 80-1,- "Guidelines for Conducting a 
Training Program Evaluation. 11 This documen t > completed in November 1979 
went a long way towards fulfilling the Army's need for a systematic method 
for evaluat ing training programs for new weapon systems . The "Guidelines for 
Conduc ting a Training Program ^Evaluation" (Harless, Note 1) henceforth to be 
referred to as the Harless guidelines , divided training program evaluation 
into ' five phases . In Phase 1, plans are made for conducting the TPE. During ^ 

"" this p tia s e7 ~aTT~~ t h e Tsackg round i n fb rmat ion , training materials, evaluation ' * 
instruments, and task documentation needed to perform the TPE are assembled 
and*. a plan for %he conduct of the TPE is prepared. Data for determining the 
effectiveness of the training program "is collected' during Phase 2, Data are 
collected to ^escribe the actual training process , testing process, training 
environment, trainee characteristics, instructor characteristics, and instruc- 
tor and trainee reactions to the training. Phase 3 consists of summarizing 
and analyzing the in-coarse test data collected in Phase 2. An analysis of 
the' test data collected in Phase 3 identifies the tasks on which test. per.-, 
formance is deficient and those tasks which require farther investigation. 
In Phase 4, training modules which warrant further investigation are examined 
to determine. ±f :the evidence collected during the previous phases suggests 
that the performance deficiencies are due to training or derive from other 
causes. The findings of the TPE are documented in Phase 5. In this phase, a 
report outlining the steps followed during the TPE and the conclusions ^ 
reached, is ^JLr^t^n_j^±: Jthe sponsor of the evaluation; For Phases 1-4, the 

~~ Har less g u i d e line s provide detailed worksheets for r ecording t he required in- ~~ 
formation. Step-by-step directions for completing the worksheets are also 
provided . * 

Using the Harless Guidelines 

ARI , Fort Knox had worked closely with Harless during v thc development of 
the Harless guidelines and was aware of the potential utility of the TP^E 
approach described in the guidelines. ARI researchers init ially_ tes ted the 
usefulness of some of the observat ion worksheets on institutional t raining 
that is routinely conducted at the Armor Center. The guidelines werjj^jpilo ted 
against Armor Officer Bas,ic (AOB) courses in supply management and forward * 
observer procedures , and courses, in. platoon sergeants' mo t ivat ion , M60A1 
track removal/ ins tallment , and mechanical training for two dif f erent machine- 
guns • From these early trials of the Harless guidelines , it was clear that 
the worksheets used* for the observation phase were a valuable aid in identify- 
ing training and -testing problems . It was also clear that the methodology 
described in the guidelines could be npplied to almos t any performance-based 
_t r^a in ing and need no t be conf ined to evaluating transition training for new 
weapon system^. ~" " ' \ — ... — 
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From ARI 1 s early experiences with the Harless guidelines, it became appar- 
ent tfyat a certain amount of skill is required to complete the various work- 
sheets in sufficient detail, and that this skill improves with practice. After- 
using the worksheets in several different? Armor Center courses, ARI researchers 
found that they became more proficient in making the^riecessary observations and 
in recording their observations on. the worksheets. To ensure that persons and 
organizations wishing to use the Harless guidelines received! familiar ization 
with and practice in using the worksheets prior to conducting any formal evalu- 
ations, ARI instituted the procedure oS conducting a s ho reworks hop on TPE 
methodology^ for potent ial .users • 

% . In the workshop, the purpose of TPE and the approach /described in the . 
guide-lines is explained. The user is then stepped through the worksheets,* 
item-by-item. When the user is thoroughly familiar with the" worksheets and 

method, he/she- theri-p^ an operational " 

three- or four-hour block of instruction. Upon completion of the practice 
exercise, the User shares observations with those of other Workshop partici- 
pants and receives feedback from the workshop leader. 

. .User Acceptance of the Harless Guidelines * - 

> - 1 " ' — ■ ■ v — ^ ; ^ 

___ The Harness guidelines were well. received by the Army and the need- for 
the- methodology described in , the guidelines Was reaffirmed by the many requests 
that ARI received for' copies of the guidelines and to conduct" workshops 'in " ' 
their use. Workshops have been -conduc ted for the USAARMC Staff and Faculty 
Training Division and^Direc toratfe of Training Developments (DTD),, the Office- 
of Armor Force Management and Standardization (OAFMS) 7 and the US Army Armor 
. and- Engineer Board. Other organizations deceiving workshops prior to the Ml 
OT-III were the Directorate of Evaluation at Fort Benjamin Harrison, TRADOC 
Combined Arms Test^ Ac t ivity (TCATA) at Fort Hood, and ARI researchers and 

^vilian, cpji tjraxl;ars ^ t Fort Bli s s in ronneation^ith^e-proposed^v^lu^t-ion- y 

of new equipment training (NET) for developing Air Defense Weapon Systems. Of 
these, the most frequent user of the Harless guidelines has •'been OAFMS . OAFMS 
ha& the mission of evaluating the state of training in the Armor force and has 
used /the guidelines to help satisfy its role as the Directorate 'of Evaluation 
at Fort Knox. In addition, the USAARMC DTD used the Harless guidelines to ^ 
certify the NET program and prepare . Operational v Test Readiness Statements " 
(OTRS) on the Cavalry Fighting Vehicle (CFV) OT-II NET player personnel. ' ■ 

Lessens Learned in Using the Haflesk Guidelines - ; • ■ 

Several lessons/^ere learned from conducting the workshops for the various 
organizations and in providing assistance to OAFMS during the evaluation of the . 
CFV arid Advanced NCO Course training. 'The first lesson learned was "'that some 
individuals are much more proficient, infusing the Harass Guidelines to objec- 
tively cvaluatq training than are others. Some observers tend to use the 
' Harless worksheets to record their subjective impressions of; tile training' or 
trainer, rather than performs he more : demanding task of miking the*specific ' 
dbserVheibp^^U^d for : ^y^t]ie Harless Guidelines. Other observers , viewing 
the same ^rs^rting, utilize the worksheets correctly; identifying important 
^^rjiining problems. : This -suggests tllat* whenever possible, -prospective data 
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collectors should be screened in order to eliminate those who for one reason 
or another do not use the Harless worksheets correctly. The practice exercise 
during the TPE workshop* provides an_ excellent opportunity to identify individ- 
uals who do or do not make good trashing observers, A second Wesson learned 
was that while the guidelines and accompanying workshop were usually effective 
in teaching potential users to plan the TPE, collect the data arfd j identify/ 
possible causes of performance* deficiencies,- it was somewhat less/ effect iVe , in 
training users to draw conclusions and make recommendations for training pro- 
gram changes based on the* data collected. The dif f iculties*~that potential 
users experienced in drawing conclusions and making recommendations suggested 
the need for additional guidance in the analysis and conclusion phases (Phases . 
3, A, and^ 5) of the Harless guidelines. : < 

- A-^-Xob-A-^d -for - gtrrre c z±xxg^^^L±n±j!Xg^^h^XBJfL iieTfjLcXej^cTe s ; •-— 7 ■ — - 

ARI was no't surprised that early ushers of the Harless guidelines experi- 
enced difficulties in drawing conclusions and making recommendations based on 
their observations of training. ARI had already discovered that the procedures 
outlined in Phase 4 of the Harless guidelines did not lead - typical military 
users to a clear identification of the causes of a given performance deficiency 
Additionally, the guidance provided in Phase 5 was inssuf f icient^ for an un- 
initiated user. Furthermore, Harless had not been tasked to include .guidance 

to the: user on -how— to -r-ecommend— program changes- based- on^ p er fo rmance— de£i^ — 

-ciencies; the tasking had only been to provide a method for' identifying those 
deficiencies and their possible causes. Procedures ; for correcting performance 
deficiencies was to be - the subject of 'a second developmental effort . ^ 

In the last quarter of FY 79, a contract was let to Seville Research Cor- 
poration to develop guidelines for specifying modifications to training on the 
basis of training problems discovered during the evaluation of training. The 
-f iua 1 pi uduct , en L I LJred" tf fre Lhudology f ox^orTi?cTin^~"De^ 
Programs ," was completed by Seville Research Corporation in April 198Q (Spears, 
Maxey and Roush, Note 2) . . 
t _ _ ' ... 

Like the Harless guidelines the Seville product was designed tq be used by 
ARI researchers to evaluate transition training during the Ml OT-III . The 
Harless guidelines - were to be used to observe training and testing in order to 
determine the causes of performance deficiencies. The Seville product would 
complete the evaluation process by specifying f ixes for the problems identified 
by the 'Harless guidelines . Although the contract with Seville called for a 
document compatible with the Harless 'guidelines , the Seville product did not 
■ interface well with the Harless guidelines. Further"woTk was required to gen- 
erate a methodology for correcting v def iciencies: that would be compatible with 
the TPE methodology described in 'the Harless guidelines * 



SECTION III 



PLANNING THE Ml OT-III TRAINING EVALUATION. 




< Range of Evaluation Activities 



Planning for the Ml OT-III began early. . The. OT-iii was to focus on re- * 
solving the training arid maintenance issues- unanswered by Ml OT-II. Just how 
th ^. was t0 be accomplished was decided in a series of meetings among the par- 
' tiGipating organizations: the TRADOC Combined Arms Test Activity (TCATA) , * the 
Operational Test, and Evaluation Agency (OTEA) , ^the Armor Center Directorate of 
Training Developments (DTD), the Office of Armor Force' Management and Standard- 
ization (OAFMS) , .the TRADO.C Systems Analysis Activity (TRASANA) > and ARI. 

* * n . tne early meetings of the above mentioned o^^nT^a tToris ~f ew % 3eci~sTons 

were reached. The meetings served primarily as a forum fo'r each organization 
to, state its views and data needs to the o ther . part ic ipant s . For example, in 
the 29 March 1979 meeting, OTEA stated its preference for obtaining user per- 
formance data as the measure ofj training effectiveness, while TCATA argued for 
a more direct measure of training effectiveness. OTEA suggested that the OAFMS / 
TRASANA/DTD/ARI training effectiveness analysis be accepted by^TCATA, but TCATA 
insisted that it preferred to conduct its own TEA. Among the other issues 
discussed during the 29 March meeting was whether the OT-III should be conducted 
at Fort Knox or Fort Hood . ^ . ' 

In 'June 1979, OTEA drafted a test plan outlining the scope of the Ml 
OT-III. The OT-FII -was to be a three-phased 'test conducted by TCATA from June 
1980 through April 1981. Ml tanks were to be delivered incrementally, begin- 
ning with three in June and continuing until 3' total of 35 had been issued by 
the e n d of November 1980.^ J)uring' Phase I (transition t raining) >_..a.-tLank- bat- — - 
talion would be transitioned company by company from the L M60 series tank'' to the 
Ml tank. Transition training, to be conducted at Fort Hood by the Ml New 
Equipment Training Team (NETT) , would consist of ^an orientation Aov staff per- 
son^iel^ individual and collective tank crew skill training, and organizational 
maintenance personnel training. Training for DS/GS maintenance personnel was 
to be given at the US Army Ordnance School at APG, MD. Phase II, consisting • of 
a live firing exercise under simulated combat conditions integrated with maneu- 
ver exercises, would be used to determine the mission and system reliability 
for the test. Phase III, would involve up to a tank battalion task force en- 
gaged in non-firing exercises against an aggressor force (up to brigade size). 
Data would be collected over a series! of field training exercises to include * 
of fen sive, defensive arid retrograde operations and would address the issues of 
logis tieal supportable "training, and f ightability . 

Th£ scope of the OT-III was further defined in a memorandum from the \i 

TRADOC Systems Manager for the Ml to 0A*FMS, DTD, ARI, Directorate of Armor Doc- 
trine and the Directorate of Combat Developments . Wilfh regard to training ef- 
fectiveness, the memo stated that the Ml NET team would conduct individual 



Fort Hood was later cfiosen iis the site for the OT-III. 



training for MOS 19K/L tank crewmen* collective training for Mi crews through 
platoon gunnery ; qual if ica t ion - 9 and organizational level training for MOS 45E 
and MOS 63E mechanics. Tactical training would be conducted by the unit ; The 
• effectiveness of the -proposed programs for individual and collective^ training, 
to' include tactical training were to be evaluated; Effectiveness of" the pro- ; 
gram of instruction (EDI) - 9 training aids, training devices and training liter- 
ature would be assessed;. Criterion-referenced performance testing conducted 
at the end 6f each 'block of instruction during the training and- at selected 
points during the test were 'to be used to determine the effectiveness of the 
training program. - 1 J _ - 

ARI .wrote a memorandum ifn response to the memorandum from t^e Ml TRADOC 
Systems Manager. ARI pointed out that the data collected on the^ adequacy of 
the POI, training aids, training literature, etc. must be .detailed, to be use- 
ful, but that TCATA -had not yet' allocated the resources to collect detail^ 
training data. ARI stressed that end-of-block performance data alone is not 
an adequate measure of performance since only a small subset of tasks trained 
are actually tested on the end-of-block tests. It was suggested that 'the re- 
sources to conduct a detailed evaluation be put at the disposal of _ DTD. 

Between 6 and 10 August 1979, a series of meetings was held between TCATA , 
DTD, the Ml NET- team and ARI.. The primary topic of discussion was "what data 
should be collected for the test .battalion, who. would collect the data, and 
who would be allowed access to. the data. DTD had been tasked by tfie CG, 
USAARMC to. develop diagnostic tests for determining the readiness posture of 
the 'test battalion on the M60A1 tank prior to the beginning of transition 
training on the Ml tank. It- was decided thdt USAARMC would administer the 
diagnostic tests and score them. TCATA agreed to collect demographic data on 
the players: and to collect hit/miss and engagement time data for the firing 

~ex~drci s~e~s~." "Th e "is su e''or^arayencies "s hou td ~fre al lowed aeeess-ty thc-da-ta — 

generated during the OT-III was raised but was not resolved during the meeting. 

Outl ining Data Requirements for the Evaluation . . v+1 

Subsequent OT-III planning meetings in wt\ich ARI took part were concerned 
solely with training evaluation. The first such meetings took place during the 
period 27-30 November 1979 and involved representatives of TCATA, ARI, TRASANA, 
and DTD. The primary topics of discussion were the role of each agency in the 
evaluation, data collection requirements and methods, and data reduction. It 
was agreed that TCATA would coordinate, monitor and control the data collection 
effort for Ml OT-III. Each of the participating organizations out 1 ined • their 
data collection requirements for the test. ARI surfaced the need to observe^ 
training and testing directly in order to collect observations about the train- 
ing and testing process, and the requirement to obtain ind ividual ~sold ier per- 
formance data on the tests administered after each block of instruction. ART f s 
data coll ection requirement.'; (which were limited to data on the TPK process; 
not on the evaluation of - the Ml tank system) were included in USAARMC \s data 
collection plan, which was drafted unchanged into the TCATA Training and Human 
Factors Data Requirements. TRASANA stated that it would need the following 
pre- and post-diagnostic tests in order to justify its participation in the 
OT-III evaluation: 1.) an SQT typo tost of all M60A1 crewmen and mechanics who 
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would be undergoing transition training, 2) an SQT type test given tb\all 
rmechanics and crewmen on the Mi at the completion of transition traihffis» an<d 
35 ajS SQT type test as described in 2) above but .given at the end of Phase III. 



.Planning the Data Co Heat ton 

; : r — — * . ) 

__■ • o t 

„ i During the November meetings, the group reviewed the roster of scheduled* 
events for Phases ,1^ II, and III to determine when arid how the data could be 
collected^. It. was decided that every' training event should be monitored by a 
data collector.- For this -to occur ,; the presence 61 one "dat<a collector per >1 tag$c 
would be tequired during each training event. plough the bulk of the training 
data would be" collected by these tank data collectors, Test Team Evaluation/ 
Supervision personnel from OAFMS , DTD, or ARI would' supervise the data collec- 
tion, collect datavas needed,' and analyze the. data provided by .the tank data * 
collectors. ° - • 



Having made some determination about whap data to collect and when and how 
it might be collected, the various agencies; divided up the responsibility for 
the design and development of -the data collection forms. ARI agreed to work 
with USAARMC in preparing data collection forms for the observation of training 
and testing. TRASANA agreed to provide demographic data collection forms, arid 
DTD, USAARMC accepted responsibility fo ride signing interview and questionnaire 
data collection instruments. TCATA agreed to provide some special forms for 
recording collective training data. It was agreed that ^the data collection 
instruments developed by tfie'various agencies would be debugged during the 
five-tank Low Rate Initial Production RAM test to be conducted at Fort Knox 
.prior^to the beginning of the OT-III. ; . ; 

' The next series of meetings occurred during the period of 7-li January 

-1933 s. _Thc ^T\ar t ic ipan t s were the same as in the previous meeting, except that 

a representative from OAFMS was also present. The issues .covered reflected the 
topics of the previous meeting. The meeting began with a review of each agency 1 
data requirements to ensure thatf each data element was required by at least one 
of the participating agencies. From this review a. final list was prepared con- 
sisting of the combined data requirements of the agencies participating in the 
evaluai* ion. Work continued on the development of ^spd'eif ic data c«^4ect ion forms 
to be us-cd during the OT-III training evaluation. Each agency had brought the • 
forms * to the meeting that they thought they would need to collect the required' 
data. Under the guidance of. ARI -Fort Knox, ,t Re number ofyforms being proposed 
was reduced considerably. All agencies a^e&d to use the ARI -Fort Knox forms 
(closely resembling Worksheets A2, A4 , B4 , B5, and^ B6 in the Harless guidelines) 
for r he observation of t ihi in ing . -Having reached some a cement on the forms 
•that would be used to coUoct the data, the five agencies 'involved tnigah out- 
Lining specific plans fo-rthc organ iza* ion . and control of tlje 0T-II1 data col- " 
lection effort; the duties and* responds lb iiit ies of each agency were mitl ined. 
Aki's responsibility wa;; liini;toa i to collection and analysis bf traihLifg effec- 
tiveness data for tfie MOS 19 K/h tank, crewmen training. ] *~ ■ . 

Mai^gjcmuMi t of the j>a_tij r Ftow t 1 

hi March 1980, AH} went to Vort Hood t o .conduct a t wo-day .workshop (n ARI . 
iTK methodology for TCATA*?; Ml OT.-iii primary data col J Oct ion managers. In a 

: '<) * ' : . 
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meeting following the workshop, ARI and TCAfA devised a data • f lbw_network , file, 
and retrieval system to be ;ased for data ^handling during the d¥-III.\It was 
decided that the data collection forms ' containing the raw data would be reprb-' 
duced :aad distributed on the basis of nled-to-knbw to the appropriate persons 
and agencies. Some of the forms would to to th^_data mahagemerit team for entry 
-into the computer data base. Others wjfuia~7c ircfclate through $ne or mQre of thb 
involved agencies and end up fin' a central papCT'^H^g^^--^*' 

■ . * * ' \ 

During the Tterch meeting the data' eollect ion f ojrms tb be used daring the - 
OT-III were finalized. \ "The complete : 'set of fbjfms used during the Ml OT-III* 
consisting of eleven different data collection instruments appears in Appen- • ... 
dices C and D. The forms in Appendix C (Forms 39, 40, 41) were designed by 
ARI fo? .the purpose of identifying training deficiencies. The remaining forms 
(Appendix D) were to be used by other agencies for their own purposes . : 

Diagnos tic T e sts , % ' ' ; 1 • • ✓ : - "« s 

*V * " - > ' - ' ' . . . . 

Members of the' TCAfA Fightability Team visited Fvdrt Knox on 28 April 1$80 
^and met with representatives of USAARAC , ~ ARI , OAFMS , and, TRASANA. Among the , 
" topics discussed were pre/post diagnostic tests, the NpTT 'training scnedule and 
POI ,,data collection forms, the piiot test scheduled at Fort Knox prior to the 
TlT-III , Control of the OT-III data collection, and resource and v r«nge ;, require- • 
ments associated with the operational test (OT) . Authorization had been'ob- * 
tained by. TRASANA t;o administer" pre-diagnbstic tests on the M60A1 prior to 
transition training and pos t-d ia gnostic* tests on the Ml following transition 

• training, -During the April meeting, a liAt of tasks to be tested on the pre- 
diagnostic test and a separate list to be tested on the post-diagnostic test - 
were compiled and 'approved by the' participating agencies . y It : was decided that, 
diagnostic testffe would be administered and score^^ instructors, from the NETT^ 

* • ' - : V ' - • ■ Ai : - < 1 - 
Scheduling . - / * - 

*_ _ _ ....... *: ' ; • t 

: Much time was. devoted to constructing a revised trainiog schedule for the 
; 0T-lil. This schedule established starting and finishing _dates by company antl 
MOS for ind ividual and cbilect ive training on the Ml , andfeor pre- aqd post- 
diagnostic t*stin£. The schedule called lor the pre-d ia^nosc rc for MOS 3 5E 
turret mechanics in Company to be given 8-12 September 1980- to he followed 
by the- 'ME transition draining and the 4 5E • pos t-d iqgnoKt ic . The diagnostics 
- and transition training for the MOS B3K track vehicle mechanics -would follow 
closely on the heels of the 4 5K training- Training tor the *19L drivers, was 
scheduLed to start 13 November and for the 19K gunncf ^loader Atahk commander _ 
l<) November. A total oT four companies were to be trained and bested in, this 
way with the last company finishing tjio'ir post-diagnostic in mid-Apr i 1 1981 . 
The traiifliig and test inp t ; schcdui e was arranged at this t iftie s because of TCATA ? s 
re<iu i rements to scUeduit' ranges wejjj in advance and to 1 finalize the Detailed 
Test lMaii.< : - V * 
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Although in*imc ehah|ieh' woiilcl *be inadr- In some of the forms subsequent to 

the March meeting; the f (**iis settled upon in L li If* meeting were for all prac- 
t (Cal purposes t lie forms used ' iliir i \\\\ the I 1 I . • 

; l iiie (oiii t-onipiiiiy plan was iati'i" -dropped ih :: lavii"r ol a t hi ce company plan 

diii- i<> :\ sicker t ti.Ui i-xpert ed . iW 1 i'very sciiedul e oj t lu- Ml tank. ^ 

■ ■' . : "'; . si -> 
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Further Plans 

..Plans on how to organize the data collection effort were further focased; 
Plans* called- for the majority of the data to be collected by TCATA data collec- 
tors- with OAFMS personnel functioning as quality control monitors; Personnel 
■ from DtD, ARl-Fort Knox, TRA SANA and OAFMS would function as data analysts and 
as datav collectors .on an as needed basis. Plans were also made to have a data 
as^i^nmtHu^ros|:cr for all tra i n i nj^j^y en t s so that each person collecting data 
would know wMcH ev*ent he/she was to observe- what forms to use, and where the 
event would Qccur - 9 at least 24 hours prior to the event . • ^ N 

Pilot Test of the QT-III Data Collect ipn Forms 

At the 28 April meeting, the upcoming Ml RAM test to be conducted 19-23 
May 1980- was also discussed. it was decided that this test should definitely 
be used as a vehicle for piloting all training data collection forms to be 
used during the 0T- III. ' 

As jvlahned, data collection forms to be used during the 0T-11 1 were 
piloted in Conjunction with the Ml RAM test conducted a,t Fort Knox in May 1980. 
OAFMS scheduled and controlled the data collection activities. OAFMS was as- 
sisted iji collecting data by DTD, TCATA, arid ARI-Fort Knox. Completed forms : 
were returned by all agencies Lb OAFMS. No problems were encountered i5 using 
the djita collection forms during the pilot test;, therefore no significant, 
change's were made iii the forms as a result of the pilot test.' 

TCATA 1 s Detailed Test Plan 



Iii an August 1980 meeting at Fort Hood, AR1 got its first look, at TCATA '-s 
Detailed Test Plan for the Ml OT-IIi. The Detailed Test Plan specified what 
data would be collected, who would collect it, 5 and how the data would be dis- 
tributed once it -was collected. The Detailed T<?st Plan provided for collecting 
training data and test data for each event occurring during transition train- 
ing,, tiie kinds ot t ra in ing .data to be collected can best be seen by referring 
to the data co i i ect i§n "forms in Appendices C arid D. Fach of the training data 
collection forms was assigned a number for ease of reference. Two kinds 01 
test data wore collected — individual performance data on /end-of-h I oc k tests 
and diagnostic test data. * Vr 

A dnt a '.'co I 1 ec L 1 on team, composed of a Company team Chief Data Collector, 
three !M a t oo'&ryTcom Chief Data Collectors, and one tank Data Col lee tor per (auk, 
wus hi col Ic^efcR'lie f%VA r * r >F , MOS K'W. und MOS LOR/h traiiidng data for each com- 
puny. Tlie taiik da t a co \] i»c t or would complete Forni \H;i OVrsonnel Status Re- 
port) .iikI Form A I) (Observation of '['est Mvehts). I he platoon data col I t or 
wtui I d eoiiiji I "e t e Form'i/ (Tra in i ng Aids Data Ilhoot ) , Fiiriii V) (Obsorv.it. hm o I 
Tra i ii i iij'J t I-'ifl/iii A I (Training Fhv i ronmeu t ) , Form h? (Ntiideiit ()iies tl oniia i re) - 
oikI For in A A (Ihslrui; tor tjiit»:;t ioiiha i r e) . The coiiiji.iiiy ehiei data collector 
w« o j Id upe i v 1 e tin oMier dull ( o 1 I r ( t oi ; - re v i * w and eon o 1 I d a t c Form-. 1 / , 
W.i , J'), A\) % A\ t A\> WWW AA , and return Mjese to Dot a Control ot the Field Test 
t c liter* At D a t a ( out rol , I he 1 orm won Id he i epi oduccd nid < op i r d I t i i but ed 
\ 0 .ipproju i .il *- ugeiic li-s '('*-. y t . , AK I , MAI'Ma* J tor analysis. The or i >• i 1 1 . i I i>j r.i* Ii - 
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form would. go to TCATA's Data Redaction Branch where the data would be entered 
into TCATA's computer for additional analysis and the original raw data would 
ho placed in a master file for future reference; Score -sheets indicating end-*: 
of -block Lest performance of each individual soldier for each block of instruc- 
tion would be completed by NET team instructors. Completed score sheets would* 
be given to the Company Team Chief Data Collector, who would forward them\ to . : 
Data Control for reproduction a.nd distribution to the various agencies. In 
addition to end-oJ-bloclj testrs, diagnostic tests measuring individual perform-", 
arico *lcvei« just prior to and following individual graining on the Ml- tank 
were to be administered to 100% of the tank crewmen and mechanics participating 
in the operational test. Pre-d iagncst ic and post-diagnostic . test data were to 
be handled in the same way as the end-of -block scoresheets *. NETT ^instructors 
were to collect the data. 'The diagnostic test^data would then be passed along . 
to the Company Team Chief Data Collector, who would forward the data to Data 
Control tor reproduction and distribution to participating agencies. 



! ;> 
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SECTION- IV 




IE M l TRANSITION TRAINING -LESSON PLANS 



Obtaining the Lesson Plans : 

As part of the evaluation of the transition training program given by the 
NETT during the OT-III, .ARI was to evaluate the NET program as detailed in the 
lesson 'plans. The lcsri#T plans were also needed to prepare some of the obser- 
vation forms to be used^Juring the evaluation. In a memorandum dated 20 Feb 
79, ART formally requested that the NETT provide ARI with copies ot any MOS 
19R/L training materials developed tor use during' &he -Ml OT-III. ARI obtained 
a set of the MOS 19K/L lesson plans 'from OAFMS. 

The original transition training materials did not provide useful training 
prescriptions and contained omissions. Some of the tasks to be trained during 
the Ml OT-III were nqt covered by the lesson plans. Though there were short- 
comings in the original version of the Ml OT-III, transition training lesson 
Dlans, the NETT indicated , that a revised set of lesson plcins would be forth- 
coming shortly.- * 

Review and Initial Evaluation of Lesson Flams 

The second version of the MOS 19K/L lesson plans was obtained from OAFMS 
in April 1980. _ OAFMS had reviewed thpse lesson plans and had suggested 
changes, some of which' were incorporated into the lesson plan 4 s by the NETT and 
„.s.pjqe of -which were not, — A~curspry -review- of - -t-htr~3:es son pians'by'ARI indicated 
that many of the problems noted by ARI in the original version had not been 
addressed. In discussing these 'problems with OAFMS - 9 ARI decided that a thorough 
review, of the, lesson plans was called for". Each lesson plan heeded to be eval- 
uated to ensure that^'it contained the necessary information and was internally 
consistent . In general a lesson plan should include detailed inf drmat ion on 
what is to be taught and tested and how it is to foe taught and tested. With 
these ideas in mind, ARI proceeded to evaluate each lesson plan* in turn* 
rioting any problems that might require corrective action. 

The evaluation quickly bogged down because of the tedious nature of the 
task. Each lesson plan seemed to tome with its own unique set of problems* 
and each separate problem seemed to require a different corrective action. 
This made 'it difficult to discern where'the evaluation for a -particular lesson 
plan should begin ^attd where it should end. It soon became clear that a more 
Systematic approach to the task was needed. 

• . K - 
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^A Systematic Approach to "Lesson Plan Evaluation 

A set of criteria for evaluating lesson plans was developed and applied 
to the MOS 19- K/L lesson plans. The use of these criteria greatly simplified 
the &task of evaluating the lesson plans.- By using the criteria,, the evaluator 
was able to focus on one aspect of the lesson plan -at a time, whl¥e ensuring 
that . all important^ aspects of the lesson plan were evaluated. The criteria de- 
veloped for evaluating the MOS 1-9 K/L lesson plans are listed* in Appendix E. 
Criteria are listed for evaluating training objectives, training procedures, 
and end-of-block' tests. * - 4 



The evaluation of the MOS 19 K/L lesson plans produced 43 pages of com- 
ments and suggestions regarding lesson plan deficiencies, ARI f s comments and 
suggestions concerning the lesson plans were attached to a memorandum dated 
1 August 1.980 and sent to the Director of Training Developments (DTD). Follow- 
up conjtacts with DTD indi cated that DTD had received the lesson plan evalua- 
tion, but that the evaluation had not filtered down to the personnel who V 
developed the lesson plans: and who were responsible for making any changes in 
the lesson plans. It was not clear whether any of ARI ' s comments or sugges- 
tions had been acted upon; 

Revised fcesson Plans 

In September 1980, DTD released the MOS 19 k/t lesson plans that were to 
be used in training the first company during the >Mi 0T- III. ARI obtained 
copies from the NETT after learning from 0AFMS that the lesson plans had been 
completed; This version of the MOS 19 K/L lesson plans was more Complete than 
previous versions and had eliminated many of the inconsistencies identified 
during ARl's evaluation.^ However many of the problems identified in' the ^val- 
uation still remained. Many of the training objectives did not accurately 
spectfy what the soldiers would be trained to do. Much of the- guidance pro- 
vided to the instructors on how the training was to be conducted was vague, and 
many of the tests designed to measure the soldier's ability to perform the tasks 
in the training objectives did not mirror the requirements of those objectives. 




It is not at all clear whether the elimination of the inconsistencies 
resulted from ARI 1 s evaluation of the' lesson plans or were the result of DTD's 
own internal review process. 



SECTION V 



DESIGNING THE TPE WORKSHOPS AND JOB AIBS * 
FOR THE Ml dT-III EVALUATION 

The August 1980 irfeeting was the final planning meeting prior to the begin- 
ning of the Ml OT-III. At this, point most of the^deXaTl^^T^tlii r"opefS£io51I 
test had been worked out. The training schedule was firm given that tanks 
were delivered on time*, ranges had been reserved f or[ the test, the detailed 
plan for collecting the data had been developed ,) responsibilities of the various 
participating agencies hac} been established, the data collection forms had been 
finalized , arid personnel for collecting arid analyzing the data had been com- * 
mitted to the task. Workshops had also been scheduled for training the Ml 
OT-III data collectors. 

All data collectors par t icipat ing in the Ml OT-III were required to Com- 
plete a workshop in order to be trained on data collecting procedures and to 
become familiar with the training evaluation forms. In contrast to previous 
workshops, very little time would be spent on the planning and analysis aspects 
of training program evaluation. Planning the TPE and analyzing the data col- 
lected would only be discussed briefly in order to help the data collectors 
understand the reasons for collecting the data. 4 Instead of the 18 worksheets 
v from the Harless Guidelines covered in earlier workshops, drily thr^fe data cbL- 
lection forms (39, 40, & 41) would be covered. 5* Whereas the Harless Guidelines ^ 
were used as both a training aid arid -a job aid in earlier workshops, the Guide- 
lines were not appropriate to use in the OT-ITI workshops. Not only did the 
Harless Guidelines provide far more information than was needed, but the Guide- 
lines were written for a user having some familiarity with educational tech- 
nology. The_ soldiers who would collect the OT-III data were naive with regard 
to educational technology. .; 



.» tyhat wa needed was a job aid designed,. specifically for individual sv^un- 
sophisticated in educational technology whose job it was to collect trainifrg 
evaluation data.. In response to' this need, ARI developed an observer's job 
aid (Witmer, Note 3). The observer's job aid borrowed tnucf^ from the Harless 
Guidelines^ but the language used was simplified with many technical terms 
being eliminated ^ and the scope was narrowed to focus on those activities per- 
formed by the data collectors Like the Harless Guidelines^ data collection 
forms were included* in the job aids The job aid described how the forms 
should be completed, including an explanation of each item on the forms. 

The forms described in the observer's job aid were to be used to collect 
in forma tig n about the Ml v transition training program. Data Would be collected 
on the training environment, the training process and the testing process • 
This data would be used in conj unction with_the performance results on the 

'. . < \. 

5" • ' 6 - 

A fourth form for determining if the training given conforms to the 

train ing planned as described in the lesson plans was scheduled to be covered 

in the workshops, but because the first group of data collectors to use the 

form did not find it useful, the form was not covered in subsequent workshops. 

* ✓ « 
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tests given at the end of ^each block of instruction to identify training-caused 
performance deficiencies in the Ml transition training program. The training 
analyst (an ARi researcher) would identify the training deficiencies and make 
recommendations to the NETT on how the- training program, should be modified _tb 
eliminate the deficiencies. : ;Tb assist the training analyst ,_ in making modifi- 
cations to the training program based bri # the deficiencies identified, ARI de- 
veloped a j ob aid fo r modifying ineffective-- or irieff ien t- t ra in ing ( Kr is t ianSen , 
Note 4) . . This training modifications job aid was designed to be* used with the 
observer's job aid in evaluating the Ml transition training program. 6 



6- / ,: 

The sections on practice and feedback were based on similar sections 
in the Seville product mentioned earlier in this paper. 

: ( ; - 16 
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SECTION VI 
FIELD TRTAT, AF THE TPE SYSTEM 



Scope and Purpose of Field Trial 



— ART^s— p u r p o s e— i n— e va luat-ing^he— Mt- OT— -Hi- -t-r An* itiori* training program was 

H.6 field test the job aids arid data collection forms developed by ARI for evalu- 
ating, training programs. The objective was not just to determine if the TPE 
methodology worked, but to demonstrate that it can be used by Army personnel 
to evaluate the training program for a major developing weapon system. In 
order to accomplish this objective mid- level noncommissioned officers were 
trained to be data collectors . TPE workshops were held to train turret mechanic^ 
(45E) , track and vehicle mechanic (63E) and tank crew (19K/L) data collectdrs. 
Separate workshops were conducted to train each of three sets of tank crew data 
collectors for each of the three companies undergoing transition training. The 

^workshops were conducted for each set of data collectors just prior * to their ~ 
participation in the operational test . Table 1 lists the TPE training sessions 
conducted for the Ml OT-III data collectors. During the workshop each data cql^ 
lector received a copy of the observer 1 s job aid for use -in the workshop ari<y oh 
the job. * 

_ o _ • __ 

~t The data collection tearn^ consisting of a company data collector^ three 

platoon data cbllectbrs^ and brie tank data collector per tarik^ was supervised 
by the TCATA Fightability Team (a designator for that group in TCATA respbria 
for evaluating the NET program). The day-to-day activities of the tank data 
collectors were controlled by the company chief data collector. The/ TCATA -- 
Fightability Team had little interaction with the tank data c o 1 1 ec t o rV*J^ T he i r 
main contact with the data collection team was through the company data col— N 
lector and came ,only when the dat,a forms reaching" data control were : improperly 
completed . In addition to the military chain of command supervision and the 
job aids, data col lectbtrs were periodically ; c&tlfected by the ARI data analyst 
or by OAFMS personnel to assist .them with any ^r&blems they might be having in 
completing the evaluation, forms. 




ble 



TABLE 1. TPE Training Session^ Conducted for Ml OT-III Data Collectors 



Workshop Training 
Dates 


Workshop Participant's 
Specialty 


Number of # 
Soldiers Trained 




16-18 Sep 80 


turret mechanic 


6 




8-10 Oct 80 


track vehicle or 


:■' 15 






turret mechanic 






21-23 Oct 80 


tank crewman 


20 




14 Nov 80 


tank crewman 


5 




1-3 Dec 80 


tank crewman 


17 ;V * 




23-26 Jan 81 

- L- 


tank crewman 


17 ' 





For purposes of field testing the TPE methodology and job aids, ARI was 
concerned only with phase 1 (transition training) ^of the Ml OT-III. Phase I 
began with the administration of tne prediagnostic (est. 4 The prediagnostic - 
test was designed to determine if the soldiers to be trained on the Ml 



i 
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possessed all of the prerequisite skills oh Che M60A1 prior to^urideVgoirig _trari- 
sitioh training. Following the prediagribstic ^ soldiers were gi^eri individual 
training oh the Ml tank. After individual trainings mechanics were_ gfven a » : 
post— diagnostic test. For tank crewmen - 9 _ individual traihihg was followed by V . 
collective traihihg which in turn was followed .by the post-diagnostic test. 
The post-diagnostic test consisted of the same tasks bh. the Ml that were tested 
bh .the M60A1 during the prediagnostic test plus some additional tasks selected 
from the transition training program. The tank turret mechanics (MOS £5E) were 
the first to be given the prediagribstic arid to undergo transition training. 
Individual training for the turret' mechanics began in late September 1980. 
Next to be trained on the Ml tank were the track/vehicle mechanics (MOS 63E) ; 
individual training for these mechanics ^ began in mid-October^ Three tank com- 
panies received _ individual arid collective tank crew training in successive 
presentations of the MOS 19K/L transition training program. Changes were made 
following each successive presentation of the transition' training so that the 
training received by the second company was. riot identical to that received >by 
the first company. Arid the third company's training differed from both of the 
companies preceding it. Changes were riot extensive \- 9 however ^ arid did riot in- 
volve changes in time or resource allocation. The training schedule for each 
of the three tank companies is listed iri'Table 2. 

TABLE 2. Phase 1, Ml OT-III, MOS 19K/L Training Dates 



V 





Dates 


Event , f 


Co #1 


Co #2 


> Co #3 


Individual Training Begins 
Individual Training Ends 
Collective Training Begins 
Collective Training Ends 


12 Nov 80 
3 Dec 80 
A Dec 80 

20 Dec 80 


5 Jan 81 

22 Jaft 81 

23 Jan 81 

6 Feb 81 


10 Feb 81 

27 Feb 81 

28 Fefe 81 

12 Mar 81 

a — - 



ARI-trained data collectors ' completed TPE data collection forms during 
the training of 45E_and 63E mechanics, as well as during 19K/L tank crewman 
training for each of three companies. During the 19K/L training, an ART. re- 
searcher was present on site to /analyze the 19K/L data as they were collected. 
Because ARl's time on site was limited arid because the data available from the 
19K/L training were considered adequate for testing the usefulness of TPE, ARI 
chose not to analyze the 45E and 63E data. In accordance with an agreement 
made by ARI. with OAFMS and TCATA prior to the beginning of the operational 
test, OAFMS and TCATA utilized . the 45E arid 63E data in satisfying their respec- 
tive missions. Examination of completed TPE data collection forms from the 
mechanics training courses and feedback from OAFMS and TCATA indicated that 
the completed TPE data collection forms provided considerable information re- 
garding mechanics training. 

As input tap the analysis of the effectiveness of the ±9K/t training^ ARI 
used TPE Forms 39, 40, and: 41 (see Appendix e) completed by the tank data col- 
lectors, the 19&/L lesson plans and the scoresljeets fwm each 19K/L block of 

tr» 
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instruct: Jon-; Additional input to the analysis came from ARI's own observation 
; °^?9 m ?_°?_ the _^^^ L transition training and through observations made by 
representatives, other agencies involved in the Ml operational: ; 

tfest. Neither da ^ 63E) training hbr diagnostic test re- 

suits wen** used as input to-' theSPiEi analysis.. ■ , , 

• ■■* ■ ' . ' • 1 . ':' •■c'--'-:-- v- v. ■ .'• ■ ■ 

Conducting, t fre, L E let d -Trial , / 

The field test of , the TP£, materials vas ©^formed in conjunction with pro- 
viding feedback to - the NET Ceam^on the ef f ectiVe^Ws : iof the Ml transition 
training. Thg -method used for ISfe field test J?as.iea : il : y\consis ted o^ collecting 
training and P er £°5 m ance data,, summarizing ^nd analyzing" fhajt/.data , making 
^recommendations for ^changing \ the . training; bafeeif on the analysis, and determin- 
ing the effects-of the changes » ma 4e; The procedural ' steps used /^n .field test- 
ing the' TPE 'material's are outlined bellow. . The* steeps Outlined were repeated for 
each lesson or^ block of tas true t ion in the individual' training portion of. the 
MOS 19K/L transition training program. ' 



Data on the training -environment . (Form 41) , training process (Form ^39) , 
and testing process (Form 40) were collected' by ART- trained NCQ tank data col- 
lectors. ? Completed forms were checked by the company chief data collector for 
Omissions or inconsistencies and then returned to the Field Test Cent'er for '» 
copying arid distribution. Data on end-of-block individual. : test performance ' 
were recorded on score s'heets by NETT instructors arid given- to the company - 
chief data, collector. The company chief data collector foirward^d the ^original 
score sheets to the Field Test Center fpr reproduction and distribution. 



At the Field Test Center the ARI training analyst received Sbpies of com- ' ' 
pleted Forms 39, 40, and 41 and the score sheets as they* came back to the Field 
Test Center. -For ' training conducted on the\ank, the. ARI analyst received data 

* collection Forms 39, 40, and 41 from each of \he 13 tanks (the. number of tanks 
in each company and, hence, the number of - trailing sites, or s tatio'ns, *per com- 

' pany) . For classroom training, fewer forms were- received for each "block of ' 
instruction. . ' 



For each block of instruction, the analyst summarized the data^ap^earing 
on the data collection forms. All the data recorded on Form 39 for a given 
block of instruction were combined " on% single summary data worksheet . Simi- 
larly the training analyst prepared summary data worksheets for Forms 40 and 
41. vThe scoresheets for each block of instruction were reviewed t.oVobtairi the 
percentage of soldiers' receiving NO-GO 1 s for each task an^ subtask. Tasks or. 
subtasks. for which 20%. or more of the soldiers tested receivea a NO-GO /an their 
first trial were considered tp represent performance deficiencies '(the. standard 
P^Y*^?^ ^y USAARMC) • *_ Pos ?ikie causes of these deficiencies were identified ■ 
from the training and testing data "recorded on Forms 39, 4i5',' and 41 J From 
these causes, the analyst, with the helpof the modificatic^is \ jhb aid/ sug- 
gested changes" to the block of instruction for eliminating the •performance . 
deficiencies. ; - ' ■ • 



^This departure from the procedure that was planned has the advantage of 
providing' training data from .each training station (i.e., each tank). 

■ ' 19 - h 
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For some blocks of instruction^ review of the scbresheets did hot turn up 
any performance^ deficiencies. Nevertheless the testing observation fbry (Form 
AO) i was reviewed'' to determine if : there were : : ahy irregularities in the testing : 
procedures. If any problems were observed in the testing process that could 
affect the validity of the test results^ -changes were suggested to the .testing 
process based on guid aire e pro vided i h the mod-if i cation s j bb aid. _ 

The training analyst summarized his findings separately . for each block of 
instruction in a memorandum to DTP,*' The memorandum identified the tasks and 
sub tasks fa£ which performance deficiencies were found and specified changes to 
the training program to correct the deficiencies. The memorandum also identi- 
fied problems with the testing procedures .and suggested changes as appropriate. 
Memoranda were usually forwarded to DTD within one day of the data Ere coming 
available for analysis. ."^V 

Changes in the transition training program made by. .the NETT were Subject 
to the approval of DTD. Some of the changes were generated by the -NETT itself 
and others were 'suggested by-DTD and OAFMS. Cha.$iges suggested by ART were in- ' 
corporated into the transition training program through DTD since ARI was not 
•allowed, by agreement with DTD, to interface directly- with, the NETT. ARI was 
informed about changes made by the NET team in the transition training POI 
by DTD . . . . v 

* _ ^ , : ■ . V - : . .. . . ... ; : 

Fie Id Tri al . Assessment of the TPE Sygtgtn * 



To determine if changes made in the transition training P6I had an effect 
,dn the quality of training; and test performance^ the data obtained before 
changes were made were compared to the data obtained subsequent to the changes . 
being made. For the >fi 6T-III this entailed comparing, the data obtained ,<f or 
each successive iteration of training (i.e.,- Company 1 data* were compared to 
Company 2 ,. and Company 2 data were, compared to Company 3) y two kinds of data 
were compared. ' The £16-66 rates for performance on the test given after each 
block of instruction were compated for each task. -Data dollected during train- 
ing and. testing using Forms 39, 40, and 41 were- compared to determine if the 
changes made in the transition training . were reflected in the- data recorded on 
the evaluation forms. Specif icallyi the number of comments: made'for each item 
on the forms were compared* from one company to the ne^ct.8 ^ *. - . « 

The field trial was conducted in order to answer a number of questions re^ 
garding the usefulness and effectiveness, of the TPE system. - Among the questions 
to. be answered were the fallowing: . ; 

Can the TPE methodology be used by the Army to evaluate the training 
program developed for • a major weapons system? 



o _ ...... 

Comments recorded on the- evaluat ion forms generally refer to training or 
tes ting problems and thus may serve as a useful* ir\dex of training effectiveness , 
The number and quality of comments also serves as a rough, measure of data col- 
lector prof iciency . , y. t ^ '.„_.■ ' 

,• ■% ■ 1 • "•■:.*■' ' 
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• . Can mid-level NCO's *e k trained tq; collect evaluation data- given only a 
short /workshop in TPE •methodology aj.jl the guidance provided by the ob- 
: ;- 3?tver f s : j bb aid? : : \ . ; 

; Can a training analyst familiar with the TPE methodology identify 
: L - training program deficiencies and their causes from the data recorded 
- ;>y oh the TPE forms by NCO's? 

V- - _ - : 

Can the training analyst make recommendations fot modifying the train- 
ing program from the deficiencies identified that are both convincing 
.and in a form that can be used by the training developer? Is the modi- 
v f ications job aid; useful in specifying these recommendations? 

. • When changes are made in the training program, is the TPE methodology 

sensitive enough to detect thes^ changes? • 

t* 1 j * 

• Do changes suggested- through -the TPE process reduce the NO-GO rate on 
the testis given at the end of each block of instruction? ' 

% . ■ How should the TPE system, including the data collect ion forms and job 
aids, be altered in order to, increase the usefulness and effectiveness 
of the system as an evaluation tool? 

The procedures used in collecting data pertaining to each question will 
be described. The data collected will be reported and the conclusions drawn 
from the data will be discussed. It should be noted that the types of data 
that could be collected during the field test were often limited by externally 
imposed constraints, forcing ARI to rely to some extent on indirect evidence. 
Because evidence was lacking in some cases, not all the questions posed were 
answered conclusively • - v 

J^J^fe t ^_ Ann y^°. use the TPE system in evaluating the, training 
program for a major weapons system was the central issue to be resdlved during * 
the Ml OT-Sii; the Army's need for an Evaluation methodology such/ as' the TPE 
system had been established previously as evidenced by th£ interest that many 
organizations had shown in , using the system. With ARl's a&sl^stande these or- 
ganizations had used the TPE data collection forms- in evaluating training pro- 
grams with some success. However , the programs to which the TPE system was 
applied prior to the Ml OT-III wete much narrower in scope, and because the 
purposes of these earlier evaluations were limited, the TPE system was' not 
fully exercised.- The first opportunity to assess applicability of the TPE 
system to the training program for a major weapons system using Army personnel 
came during the Ml OT-III. 

Information was drawn from several sources in assessing the Army's ability 
to apply the TPE system teethe evaluation of the OT-III transition training 
program. One source of information was AEI f s observations regarding- the plan- 
ning process that preceded the Ml OT-III., As described before, the sdope of 
the OT required extensive coordinated planning by several organizations' includ- 
ing ARI. ARI f s role in the planning process was to ensure that ,the TPE system 
was- incorporated into the overall test plan in such a manner as to fully utilize 
* •"«"."■ 
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its potential as an evaluation instrument given only those Army personnel arid 
resources that could be made available for the evaluat ton "of the training pro- 
gram. TCATA^ who controlled the data collection effort Jy^as able procure 
the personnel arid resources required to utilize the TPE sy^eiri^ arid the TPE . 
system was incorporated into the detailed test plan in a way that was icceptabl 
to AR1 and all of the other agencies involved in the program evaluation. This 
demons tra ted that the Army could incorporate the TPE system in an /overall test 
plan for evaluating the training program for a major weapons system without 
compromising the TPE system or the overall test of which it is a part . 

J 

A second source of information regarding the Army 1 s ability to use the 
TPE system was the Army implementation of the TPE system during the OX. Among 
the important factors in the imp 1 emen t a t ion were: the Army's capability to" 
follow their own plans in using the system; the timeliness with which the 
evaluation data were collected, forwarded to the- training analyst, and acted 
upon in accordance with the analyst 1 s recommendations ; the quantity and quality 
of the data collected by the NCO's using the TPE worksheets; and the extent to 
which the data were used by the various organizations. 

s x -\ . ; _ 

The manner in which the TPE system wast used was in accordance with the 
detailed test plan with two notable exceptions. The task'of collecting the 
TPE data was nob shared, by the platoon and tank data collectors as planned. 
Rather the tank data collectors were required to shoulder almost all of "the 
data collection task, including certain data collection requirements _that were 
added after the OT-III began. Some evidence suggests that the P er i Qrnianc ? _°f 
the tank data collectors was adversely affected -by the large number of data 
collection forms they were required to complete^ ^ n ^ e P? n dent observation of 
some blocks of instruction by the ARI analyst and ^valuators from other organi- 
zations indicate that the quantity and quality of the data collected by some 
of the data collectors was less than might reasonably be expected. Informal 
conversations with some of the data collectors also suggested that the number 
of data collection instruments to be completed had a negative impact on the* 
motivation of the soldiers to conscientiously record their observations on the 
TPE data collection forms; The problem was exacerbated by the conditions under 
which -the data collectors- were forced to .work^ Data collectors frequently 
were required to collect data all day in wet or cold and windy weather • Often 
they were called upon to collect data right through (and long after) normal 
meal hours. Many of the "creature comforts" provided to the participating 
units were not given to the data collectors^ and they were treated as uririeces^ 
sary by " the NETT and by the units being trained. A second deviation from the 
detailed test plan that adversely affected the evaluation lef fort was that - 
changes to the training program were not made on a day-to-day basis as planned. 
Plans called f or ^recommendations for changes in the training program to be re- 
ceived by the persons responsible for making those changes within 24 hours of 
the time that the class being evaluated was conducted. Because of problems 
'unique to the Ml OT-III at Fort Hood ARI did not receive the data collection 
forms in a timely manner and the persons responsible for , making the changes .' 
did not receive ARI T s recommendations for changes until weeks after the train- 
ing had occurred. Upon receipt of ARI 1 s r ecommendat ions , the NETT made some of 
the changes 'suggested by ARI and did not make others. Unfortunately, the NETT 
did not document those changes that were made in the training program, thereby 
leaving some question as to the extent of the changes made. 



Despite somc.dif f icai^ijs in using the TPE systefti, ail of the Army agen- 
cies involved in the evaluation of the Ml training program used the data gener- 
ated by the TPE system for their respective purposes. The primary users were 
OAFMS , TCATA , and ARI. 6AFMS used the data for certifying; the readiness of 
theOT^iii players as Ml qualified crewmen and mechanics; 0AFMS also used the 
delta in certifying the effectiveness of the Mi transition training package. 
JTCAtA kept detailed records of all the TPE training data and used the data and f 
memoranda generated by ARI as input for their own independent training effec- 
tiveness analysis. DTD and TRASANA also' used the data in conducting a Cost 
arid Training Effectiveness Analysis (CTEA) on the Ml transition training pro- 
gram, although TRASANA relied^heavily on pre- and. post-diagnostic test data in 
its analysis. Many of DTD T s directives to the N$FT to modify the training 
program wcfeAased on changes first recommended by ARI. 'All of the organiza- 
tions involved in evaluating the effectiveness of Ml transition training seemed 
to find the data, useful. This is not surprising in that the TPE system pro-, 
vided the largest pool of objective training data that was available during : 
the Ml 0T-I11. 

The TPE system was. designed to be used by soldiers who were not sophisti- 
cated in educational technology and who had no previous experience in collect- 
ing or analyzing training data. The NCO T s who served as the tame data collec- 
tors*and their supervisors, having neither methodological sophistication nor 
•previous experience , were an ideal group for test ing whether typ ical Army per- 
sonnel could collect evaluat ion data given a short workshop in using the TPE 
data collection forms arid the observer T s job aid. The TPE workshops conducted 
by the training analyst provided the first_pppdr trinity for ^etermiriirig the 
ability of the data collectors to use the TPE system. As a whdle^ the group 
responded well to the training given in the workshops . Questions asked during 
training, performance dur ing the practice session^ and comments made during 
the discussion following practice indicated a high level of under s tanding for 
the majority of the soldiers being trained. While some of the soldiers 1 ques- 
tions indicated misunderstanding of a few of the TPE terms used (e.g., isolated 
practice, level of reality), the preponderance of the soldiers 1 responses sug- 
gested the soldiers could -Use the* TPE data collection forms to make objective 

observations. :\ , 

\ _ 

. • The quality and quantity of the observations recorded on the forms during ■ 
the transition • training^ however , were not what might be expected on the basis 
of the performance demonstrated during the vorkshop. Many of data collectors 
were not using the forms they were designed to be used. The forms listed 
specific items (see Appendix G) which ^required the data collectors to observe 
training to determine if it met specific criteria described in the observer's^ 
job aid -and discussed in the TPE workshop. When thesfc, criteria were not met, 
the data collectors were encouraged to record a comment describing;: what ; went 
wrongs But as the data began to come in, it became apparent that many ccf the 
data collectors were treating the TPE data collection forms as simple check- 
lists, responding to the items subj ect ively based "i6n their general impressions 
of the litems • rather than using the objective criteria specified in the observer 1 
job aid". (Witmer ; Note 3). Furthermore the numbfer of comments recorded- on the 
forms were far fewer than might be expected based on independent observations 



•of the training by the TPE analyst and others.. For tuhatelyi sufficient data 
Were obtained, through the efforts of a handful of data collectors,^ to allow 
the transition training program to be evaluated as planned. 

An ART researcher analyzed the data provided by the team of data collec- 
tors. Some of the procedures used by the analyst were designed specifically 
for evaluating training during the Ml OT-III and were undergoing their first 
real test during the OT. jhere was a question As to whether the training 
analyst coulck ident if y training program deficiencies and their causes given 
° only the .tfata recorded by the NCO data collectors on the data collection forms. 
: ; For the most part, the analyst was able to identify performance deficiencies 
\ and their causes from the data recorded oji the data collection forms/. In a few 
; cases the analyst was aided in ^identifying causes by^ his own independent ob- . 
nervations of the training and by data froth other sdul^iss. There was-: at least 
one class for which the data recorded on the TPE forms provided no clues re- 
garding the causes of the performance deficiencies observed. In this particu- 
lar case , it was necessary for the analyst to review some additional data col- 
lected by TCATA and to question some of the persons who had observed the class 
about how the class was conducted in order to identify the cause of the defi- 
ciency. It should be stressed, however, that the analyst was forced to resort 
to search fpr the necessary information only because the data collectors 'did 
"hot record the necessary information oh the TPE forms, and was not due to the 
design of the forms themselves . . . " 

Having identified training deficiencies and their causes, the analyst was 
in the position^ to recommend changes to the transition training to correct the 
deficiencies. The analyst derived recommendations for changing the training 
prpgtam fyom the modifications job aid (Krist iansen , Note A) . The modifica- 
tions job aid was used by the analyst to specify what the NETT should do to 
correct eacb of the performance deficiencies observed. The NETT was told in 
simple language what act ions "to take to correct the deficiencies, the NETT^ 
however, was reluctant to make changes in the training program; therefore the 
usefulness to the NETT of the changes derived froni the mod if ica-t ions job aid 
was not fully determined.. Some of the recommendations made by ARI were imple- 
mented by t be NETT af terc the Director of DTD issued a memorandum reiterating 
certain of ARI * s recommendations and directing the NETT to incorporate the 
"recommendations into the transition training program, this demonstrated the 
pbteht ia*! usefulness of the modifications job aid as a source o£ information 
for specifying alterations to training programs on the basis of observed per- 
formance deficiencies. The use of the modifications job a id greatly simplified 
the/process of. deriving program changes from program deficiencies and thus made 
thV^analys t \s job much easier. 

• M- . ■ ' - ; • _' j _ . ■ ■- ' 

, r In response to recommendations made by-ARI and reinforced by DTD> the 
NETT adde^l "demons trat ions to some lessons where- there had previously been none 
and required instructors- to adhere more closely to the lesson plans . These 
changes were instituted prior °to administering the transition training for the 
second and third companies. Evidence that these changes were indeed made came _ 
from the comments.::recorded by the data collectors and werg verified by informal 
contacts with the data .collectors, NETT, and OAFMS . The number of comments* 
recorded indicating that tasks were** not demonstrated or that lesson .plans were 

" 2A 
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not followed dropped sharply from the f L r « t to the second company -arid remained 
low' for the third company. The number of comments con corning demons it rat ions 
was. 81, 11, and 13 for the .first, second, and third companies , respectively, , 
T-toe number of comments concerning adherence to lesson plans was 27, 9, and 8 
for' tlie first, second, and tlvird companies, respectively. The sharp decline 
in the number of common ts concerning dembnst rations and adherence to lesson 
s plans from the first to the: second and third companies parallels known changes 
in the training process, and thus demonstrates the sensitivity of the' TPE 
methodology for detecting changes in the training process. 

; :.. . i-u '.•••T l ;.. ... . . 

Additional indicators of the sensitivity of the TPE methodology to' program 
cliangcs were not available because some of the changes made were not documented, 
making it difficult for the analyst to verify that the changes had" actually ■ 
been made. The analyst was not informed when changes were made in the training 
•program, and the lesson plans were not revised to include many of the changes 
that were instituted. This reduced the ability of the training analyst to 
assess the effects of the various changes made during tUe transition training 
program, ' ^ • 

It was expected that as changes generated by' "ftRI * s anaiys is of the train- 
ing program were implemented by the NETT that the percentage of tasks for which 
soldiers received first-time GO* s would increase. Such an increase would sug- 
gest that the changes made in the training program asthe result of Jjie TPE 
analysis were increasing training effectiveness. The predicted inenrase in 
performance over the three companies was obtained. The proportion of tasks 
for which 166 percent of the soldiers tested received first-time GO's increased 
from 24 percent for the first company, to 34 percent and 53 percen-t for the 
second and third companies, respectively.^ While such increases, may be due in 
part to other factors (e.g., reduction of standards for some tasks and elimina- 
tion of some task requirements) , the trend toward higher firstN-time GO rates 
constitutes indirect evidence that the changes made in the training program 
from one company to the next increased training effectiveness and thus supports 
the usefulness of the TPE methodology responsible for these changes. 
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; -r ' ,. SECTION VII 

• ■. "1 ' ..... 

% -REVISING THE TPE" SYSTEM ON- _ 

BRQB.LEMSL XBENTIFIRD DURING THE F-I££B-TRIAt ' 

The field trial of the TPE system during the Mi ©T-III indicated that the 
system could be used by Army personnel to identify training jirogram deficien- 
cies and their associated causes and to make recommendations for changes in the 
training program that when implemented reduce the number of performance defi- 
ciencies observed. -But thd field triaS also identified problems in using the 
TPE system to evaluate large scale training programs such as the Mi OT-III 
transition training program. In the paragraphs that fellow, solutions to 'some 
'of the problems encountered in using the TPE system during the Ml OT-III Vill 
be proposed and changes made to the data *colriection forms and job aids will be. 
described and fiiscussed. 

Motivation of the Data Collectors / ...... 

Many of the problems encountered by ARI during ttie Ml OT-III arose because 
some £>i the essential elements of the training evaluation were not controlled 
-by the training analyst. The data collectors are a case in point. The data 
collection team was trained by ART but was controlled by TCATA during data 
collection activities. The training analyst visited the training site 
periodically and talked with the data collectors in an effort to determine' if 
the observers were having any difficulties in completing the data, collection 
forms. During these visits to the field, the analyst noticed that many of the 
data collectors seemed to lack the motivation to perform the data collection 
task Well..-- The analyst also observed some of the factors that were contributing 
to this lack of motivation, such as too many forms, poor working- conditions , and 
being treated as unnecessary by the NETT arid the unit. However, ARI had little 
control over these factors during the OT and thus was unable to change these 
factors. In subsequent applications of the TPE methodology, the following . 
steps should be taken in order to .ensure that the data collectors are properly 
motivated. In the TPE workshops 'the importance of the data collection task to 
the overall evaluation arid what evaluation can accomplish should be stressed. 
Qata collectors should be forewarned that they are likely to be treated as ad- 
versaries by the instructors who deliver the training, but- they should be as- 
sured that their job is as important as that of the instructors and that they 
will be fully supported in their efforts-. ' If possible, the data collectors 
should work directly for the analyst.* In ttjis way - the analyst can exert con- 
trol over the data collection activities, ensuring that the number of forms to 
be completed by any one data collector arid the number of hours spent completing 
these forms are limited to a reasonable level. Likewise the" analyst cart ^ensure 
that the data collectors are supplied with the necessary creature comforts. 

Data F low Problems and S4t gge^ted- Solution s 

* One "of the problems encountered during the Ml OT-III was that information , 
was "riot received in a timely manner by those who most needed it ^ The analyst 
sometimes did riot receive" the evaluation data- until several days after they, 
were collected, and the NETT did not receive the analyst's recommendations' until 
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several wee^ after the recommendations had been made. In order to ensure 

.the. timelysf lbw'6'f data from the data collector to the analyst and from the 
analyst to the persons responsible for instituting changes in the training 
■ program j a v direct line of communications should be established from the 
analyst in both ..direct ions. In order to speed the flow of data to the analyst , 
the originals oft completed data forms should go directly from the data collector 
to the training analyst^. The analyst would then analyze ?he data . and the 

'results of this analysis would be made available to other organizations. If 
other persons or organizations needed tjhe raw data, they would have to obtain 
copies .through the TPE analyst. ' The analyses recommendations for; changing 

7 the training program would be forwarded directly to a member of the program 
staff responsible for making changes in- the training program. This person 

would check into the possibility of making each of the changes suggested by 
the analyst and would inform the analyst -which changes were made arid c which 

were hot. 

* A 

Other tharv* speeding up the flow of information bet-ween the analyst arid 
the per^oa(s) .responsible for making training" program changes, o'ther advantages 
may accrue, from a direct line of communications between the analyst arid the • *' 
program staff . The f » chances for the analyst to convince the program staff to 
make the recommended # changes are increased through direct contact, as are 
the opportunities for determining why some of the recommendations were not 
adopted. Additionally, personal contact on a regular "basis between the analyst 
,and the program staff may reduce the animosit.y that tends to develop between 
the evaluator and those whose iprogram is being evaluated. 

Problems in Using the Data Collection Forms 

Perhaps the most critical problem surfacing during the Ml OT-III was 
the tendency of many of the data collectors to treat the data collection 
forms as simple checklists, forming general impressions about the training, 
and responding to the TPE items on the basis of these impressions rather 
than upon the objective criteria provided. This led manydata collectors to 
mark an item OK and record no comment when the item should have been marked 
Not OK and a comment recorded. The NCO data collectors exhibited a general - • 
reluctance to record their observations in a written comment, even when they 
judged some aspect of training to be Not OK. In an attempt to correct this 
problem, the -TPE data collection forms and the observer's job aid were 
modified considerably. 

" '. • ' . J - 

Revision of the Data Collection Forms , 

The three data collection forms - a training observation form, a testing 
observation form, and a training environment data collection form - used 
during the Ml OT-III were retained, but the formats were changed, new items 
were added, and many of the bid Items were revised. The training observation' 
and training environment forms used during the OT required the data coiiectors 
to make OK-Not OK judgments for ench 'aspect of the training environment or .'; 
training process that they were being asked to^observe. Requiring OK-Not, OK , 
judgments, encouraged data col lectors' to rely on general impressions rat her'; than 
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on objective data;. Therefore, OK-Not OK judgments, were eliminated from the 
revised data collect ion^forms'. Ail of the items on the revised £otm&, at^ . 
phrased in question format so that the observer can address each item with a 
"Yes" or "No" response. The wording of the items on the revised forms is 
g^ite specif ic, leaving little room for interpretation; The increased 
specif icity of the Irtems Encourages, the observers to attend more closely to 
th£ training and also reduces " the. Seed for the observers to. clarify their 
responses with comments. The need for comments, was farther, .reduced by the 
addition of a number of new items. Several specif ic new items replaced one 
or two less specific old items , thereby reducing the need to ciar i£ y ..responses' ' 
"with cominents. The addition of the new items increased the power of the TPE 
system to. obtain specific information regarding such critical tr^oing events 
as demonstrations , practice, and feedback. The wording of some items was : y 
revised, not just to increase specificity, but also to make the item more 
easily understood. I-tems using such specialized terms, as need-to-know* 
nice-to-know, performance-based^ subject-matter based, level of reality, 
and isolated practice were revised. The specialized terms were eliminated 
and more widely understood terms were substituted in their place. Two items 
were eliminated because they were ambiguous and produced- no useful data -during 
the OT-III. Overall ratings of the training arid test by data collectors - 
during the Ml OT^III provided rid useful information arid thus were not included 

. in the revised data collection forms. The rating scales, tended riot to be 
useful because f$f an extreme. leniency bias bri the part of the data collectors 
arid a general reluctance to use the rating scales as they were designed to be >: " 
used* A> question concerning the approximate number of soldiers receiving No-Gb ' s 

^ provided little 'additional inf ormation during the OT arid was thus eliminated 
when the , f orms were revised. 

A fourth data collect ion- form - a Training Plan Description/Training 
Events fo $m - had been designed for determining if training is ..conducted as 
specified in the training plan. The form, listing the major training events 
and oth£t* pertinent information abstracted from the lesson pla$s, was'to be 
Used by" data collectors during the Ml OT-ftt tb record if the major training 
events. were conducted as planned. However, the lesson . plans developed for 
Mf- 'transit ioj ' training included little detail, and the data'coiiectors could 
;^see jio c icar advantage in p.lin*^ information abs t rac te^.- f rom the lesson plan 
f over using the lesson pia^ ilffieif. Because the form wjas not perceived as being 
useful by the data collectors and because xhe analyst felt that the additional 
data provided by using the^fotm would be outweighed by the additional data 
collection burden placed upon th^ data collectors, a decision was made not to 
use the form during the Ml DT-7 II. 

The decision not to use the T^rqining Plan De script! on/ T r a i n i ng Events 
Form during t lie OT- 1 L I was based on factors unique to the OT. and is therefore 
not indicative of the general usefulness of the form as an evaluation instrument . 
To encourage subsequent use ^ t lie form was revised to make it easier to use 
and more* objective. On the revised Form, the training events were listed more 
boric iSe 1 y , making it easier for the data collector tq record which of the 
t r a i ri i n eve nts occurred a rid wli i c ji did not. As with the o t he r TPE forms, t lie 
revised i orm did not call for ('K 1 Nbt OK judgments, but merely requi red tlie . 
cl.it ; i col lector to record whether or hot a particular event occurred • - 
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The Revised Job Aids - : . . > 6 * ^ >; . 

.The revised data collection If orins arid the items comprising them t are, 
described,. -in a revi^gd ; vetsi^ft of the bb server 1 s 30b aid (Witmer^ ; Note L '5> 
The Earlier" version* 6 l f the ' ■ aid _XWitiner ,_Nafce 3.) was designed 'specifically ' 
for use by the data collectors" during the Ml OT-III. *The revised 30b aid was 
written so that it could be ijsed by anyone who was given the responsibility 
to observe training arid testing for the purpose of collecting evaluation data. 
The revised 30 b aid was changed considerably 'from the earlier vers ion. Besides 
the changes to some items - 9 the addition of other items, arid the deletion of 
OK-Not.OK judgments, changes were made in the basic orgariizat ion of the job 
aid. Whereas the earlier job ^id had specified the format of each worksheet 
and . the - items appearing on. tljat worksheet, 'the revised job aid provides a 
suggested* format and lists iti'emsy .'that might be selected f bir inclusion on* that 
worksheet. This allows the training analyst the option of using, the suggested 
format or developing. ari alternative format. It also offers the /affalyst an 
opportunity, to select the* items appearing on each worksheet.. Tor assist? the 
analyst in selecting the items and to make the. job aid easier to . understand , 
items relating to th^ same type of training activity were grouped together. 
For example , all items related to practice were' grouped under the "Practice" 
h^din'g ,; _ahd all items relating to feedback were listed under "Feedback." To 
further aid. the analyst in selecting items, items were dichotomized according 
to level tof /difficulty. An asterisk was used -to identify litems 'that _ require 
more skillful observation. This enables the analyst to tailor the worksheets 
to the skill's of the observers who will be using them. 

^_ S 1 . _ 

The modifications job aid was also revised QCristiansen , Note 6) to make 
it compatible with the revised observer's job aid and easier to use. For 
each of the items for identifying training problems listed in the observer's 
job aid, the modifications^ job aid proposes possible solutions 'to the problems. 
As in the observer's job aid, the modifications job aid lists "solutions by 
worksheet and groups solutions relating to thc| same type of training activity 
together. fhe table of contents has _been expanded to allow .the analyst to 
quickly locate the suggested soiutioi^for each training problem identified 
on the observer's worksheets. The* revised! modifications job aid has been 
greatly expanded over 'the earlier version and provides mtfcri more detail on 
"how to modify training programs 3 on the basis of deficiencies identified during 

training. ' B • 

» 

As mentioned earlier a set of criteria that could be used in evaluating 
lesson plans was developed and used in evaluating the Ml transition training 
lesson plans. In conjunction with the development of these evaluation criteria 
a first cut was made at producing „a. job aid for the evaluation of lesson plans, 
this preliminary job aid for evaluating lessori plans relied heavily on the use 
of examples from the Ml transition training lesson plans^to ^illustrate the 
types of pr*of)lems that generally occur in the design of lesson plans. Following 
the Ml OT-Iir , the' lesson plan evaluation job . aid was rewritten (Krist iansen 
and Witmer, Note 7) omitting'^ the. examples f rom the Ml transition training 
and including more general guidelines for the evaluation of lesson plans. 
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The job aids described above provide guidance on how to evaluate lesson 
ipians (Kristiansen and Witmer, Note 7), how to observe training and testing 
to identify training program deficiencies (Witmer, Note 5) ~, and how to modify 
traifting in response to deficiencies discovered ^during training (Kristiansen ; 
Note 6); These job aids alone do not provide sufficient guidance to the* 
training analyst for conducting a complete training program evaluation- 
For example, the job aids do not provide guidance on planning the program 
evaluation selecting , training, and supervising data collectors, interacting 
with the program staff and other participating agencies, or procedures fot\ 
interpreting and reporting the data. To provide guidance to the training 
analyst -on these and other facets of training program evaluations not covered 
in the previously described job aids, an analyst's job aid was developed 
(Rristiansen and Witmer, Note 8) . i 

These four job aids taken together provide methods -and materials that 
can be used by Army personnel in evaluating training, programs. The guidance 
provided in these job aids is much more detailed^ than that provided by the 
Harless Guidelines which the job aids replace. Providing separate job aids 
for lesson plan evaluation, training observation , -and training program 
modification simplified the evaluation task by making it easy to locate 
information concerning a particular evaluation activity. Further simplification 
was obtained by reducing the' number of worksheets to be ^completed by the training 
observer and training analyst. The Harless Guidelines included eighteen different 
worksheets - a worksheet for almost every possible training evaluation i act ivity." 
The sheer number of worksheets to be Completed made the evaluation task appear 
quite formidable. Akl ' s experience during the Ml OT-III showed that many of 
these worksheets were not necessary for conducting a training program evaluation. 
Further examination of the Harless worksheets following the Ml OT-III indicated 
that much of the information called fpr by the various- worksheets was redundant 
and that the kinds, and amount , of information provided by some worksheets did not 
justify their existence. The revised .job aids include only four worksheets ' or 
data collection forms . . 

,■ . ■ ^ » • 

J. . -- - -- 

These data collection forms provide the necessary guidance to the training 
observer to ensure that the^t>raining data needed by the analyst are collected ; 
during training and testing. Although the training analyst may wish to develop 
additional forms for collecting some of the information needed tip evaluated 
particular-draining program, no other forms are required by the TPE system. 



SECTION VIII 

CONCLUDING REMARKS ' . y: 

The TP E" system For "evaluating the "effect iveness and efficiency of training 
programs as described in the revised job aids offers substantial advantages 
over other methodologies used in evaluating . training. Like evaluation methodjs 
that employ specially constructed post-tests, TPE provides an 'objective means 
of determining training program effectiveness. But unlike post-tests, TPE 
does not require substantial amounts of the soldiers time over and above what 
would normally be spent iri training. And while post- tests may^merely indicate 
that a given program of instruction (POI) is effective or ineffective, TPE is . 
. capable of diagnosing specif ic 'problems in the POI. Like questionnaire methods 
of evaluating training, TPE seeks to identify the source of training problems 
by gathering extensive information about the conduct of the training program. 
But unlike questionnaire methods, the information is obtained throagh direct 
observation, rather than from second-hand, post-facto accoants of the training 
program by the program participants. . ;. . 

.__ j» 

The TPE system described in the revised job aid is the product of many 
months of developing prototype data / coliection instruments and job aids and 
< -testing those materials in the field with typical user input against typical • 
Army graining. Through field testing .the prototype materials, many valuable 
lessons were learned, which were incorporated into revisions of the TPE system. 
In revising the TPE system, the job aids were refined to make them easier to 
use and comprehensive enough to provide detailed guidance for the training ob- 
server and training analyst. The revised job aids (Witmer, 'Note 5; Kri&tiansen, 
rjote 6; Kristiansen and Witmer , Note 7; Kristiansen and Witmer , Note 8)' provide 
a wealth of information for anyone who is involved in the evaluation of train- 
ing programs, and are particularly useful for persons responsible for evaluating 
training programs in the Army. 
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'TRAINING OBSERVATION FORM 



Date 



Place- 



Observer- 



Instructor- 



Time (from) 
Nr Trainees 
Subject 



(to) 



Nr AIs 



LESSON PLAN ' ' ' ;^ ■ . "^fcx 

Do we have a copy? ^ : Y 

If so, was it followed? ■ y 

Was the lesson plan changed as a result m of this instruction? Y 



CONTENT 

Were objectives stated in performance terms?? 
What were they?^ 



Y 
Y 



N 
N 
N 



N 
N 



HANDS-ON TRAINING 

Did lesson involve hands-on practice? 
If yes, were all trainees given hands -on experience?- 
If riot all, what percentage? V r 

Did hands-on supervision Appear adequate? 
Did hands-on time per trainee appear Adequate? 

LECTURE/DEMONSTRATION ) 

Did [lesson* involve lecturing? r¥~ N y Demonstration? ~ 
'DicftectUfe appear adequate? If no, what observations did 

you make that led you to this opinion? {List below) ' 
Did demonstration appea^ adequate? If' ho, -what observations 

did you make that led you to this opinion? (List below) 
Did instructor ask for feedback from trainees? 
Did he get feedback? <i . • 

Did he handle it adequately? : * 

MISCELLANEOUS " * 

I/i your opinion, was the total time adequdte/excessive/ • 

inadequate for this lesson? -' :m 
In your opinion, were the trainees trained to criterion 

during this block of instruction? 
If .no^ what observations did you make that led you to this 

opinion? 



Y 
Y 



T 
Y 



Y 

Y 
Y 



N 
N 



N 
N 



N 
N 



c;. N; 
>N 



US Army Research Institute 
Field Unit - Ft Knox 
*>Ft Knox, KY . 40121 
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APPENDIX, b; 
TRAINING OPINION QUESTIONNAIRE 



: To quality control future XM1 training program development efforts^ we 
^eed your opinion regarding the training you have just completed. Attached 

is a list of tasks (numbered J- thru ). Please respond to each task as 

follows. * 



1, Circle YES or NO opposite each task to indicate whether the task was taught 
as part of your training. « ^ 

2. ^If th^task was taught (YES), ^swer the^ remaining questions for that 
ta*St by circling the number that best reflects your opinion of the training. 



1 = Very effective 

2 ^Effective 

3 = Borderline 

4 = Ineffective 

5 = Very ineffective 



3. If- the task was hot taught 
the procedure. 



, go to the next task listed and repeat 



•After you. have rated each task according to the above procedure , complete 
the questionnaire ;b^' answering the last two questions. Be completely frank 
in making your re^onses and comments. 



NAME- 



RAN It 



POSITtUN_ 
DATE 



US Army Research Institute 
Fort Knox Field Unit* part Knox, Kentucky 



" . . t 
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TRAINING OPINION QUESTIONNAIRE 
I ■ Very.&ffective 2j> Effective _3_- Borderline Jj> Ineffective Very Ineffective 



jfas this ^ 

task . Subject 

tMSh t ? ^ I nstructor Hatter __ 



Methods 

6F Training 'Hands-on Reference Training Review and 
In struct ion Ti»e 



Practice Materials 



Aids Critique 



Hon would ybu rate tha 

_How_woyId ygu_me_ effectiveness of this 
the overall training training in preparing 
for this task? " you for your job? 



UPS before operation 
and services 
jhting the aain gun 

j the aain gun 

* 

ling prepare-to-f ire 

and_services_ 

ng targets at high 

ying targets at high 

g targets at high 

ng targets at long range 

ying targets at long 

ting targets- to gunner 

ning correct range 
ie return) to target 
g.iain-|uh targets 

he thecal sight 

g_ targets while firing 
adve - 
.rounds while firing 

aove 

g targets under 
5 conditions 
I targets using 
t fira 

ng high speed evasive 
rs fro* TCis- stat Ion 
ng battle range 

ng smoke grenades 

fire checks 

vice's 

trig TC warning lights 

ing after operation 

and services 

ing TC maintenance 

hs 

UST PACE) ' 



Yes 
"Yes " 
res 
Yes 
Yea 
Yes 
Yes 
Yes 
Yes 
Yes 
Yei 
Yes 
Yes 
Yes 
Yes 
Yes 
Yes 
Yes 
Yes 



Yes 

-!■»!, 
Yes 
Yes 



1 2 3 4 S 
1 2 3 4 5 
1 2 3 4 S 



1 2 3 4 5 
1 2 3 4 S 
1 2 3 4 5 
1 2 3 4 S 
I234S 
1 2 3 4 S 
12 3 4 5 
1 2 3 4 5 
12 3 4 5 
1 2 3 4 5 



I 2 3 4 S 
1 2 3 4 $' 



1 1 3 4 S 

1 2 3 4>S 



1 2 3 4 S 
1 2 3 4 5 



1 2 3 4 S 

12 3 4 5 

1 2 3 4 5 

12 3 4 5 



I 2 3 4 S 
1 2 3 4 S 
I 2 3 4 S 



1 2 3 4 S 

1 2 3 4 S 

1 2 3 4 5 

1 2 3 4 S 

1 2 3 4 5 

1 2 3 4 S 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 



1 2 3 4 S 



1 2 3 4 S 

123*$ 



1 2 3^4 S 
1 2 3 4 S 



1 2 3 4 S 
1 2 3 4 5 
i 2 3 \4 S 
1 2 3 4 5 



1 2 3 4 5 
1 2 3 4 5 



1 2 3 4 5 
1 2 3 4 5 



I 2 3 4 S 
1 2 3 4 5 



1 2 3 4 S 
1 2 3 4 S 



1 2 3 4 S 
1 2 3 4 5 



1 2 3 4 S 
1 2 3 4 5 



1 2 3 4 S 
1 2 3 4 5 



1 2 3 4 S 
1 2 3 4 S 



1 2 3 4 S 
1*2 3 4 S 
1 2 3 4 5 



1 2 3 4 S 
1 2 3 4 5 



1 2 3 4 S 
1 2 3 4 5 



1 2 3 4 S 
1 2 3 4 5 



1 2 3 4 5 



1 2 3 4 5 



1 2 3 4 5 



1 2 3 4 5 



1 2 3 4 5 

1 2 3 Vi 



1 2 3 4 5 
1 2 3 4 5 



1 2 3 4 5 
I 2 5 4 $ 



1 2 3 4 S 
1 2 3 4 5 



1 2 3 4 5 



1 2 3 4 5 



1 2 3 4 5 



1 2 3 4 5 



1 2 3 4 5 
1 2 3 4 5 



1 2 3 4 5 
I 2 3 4 5 



1 2 3 4 S 
I 2 3 4 S 



1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 



1 2 3 4 5 
1 2 3 4 5 



1 2 3 4 5 
12 3 4 5 



12 3 4 S 
I 2 3 4 5 



1 2 3 4 5 
1 2 3 4 5 



1 2 3 4 5 



1 2 3 4 5 



1 2 3 4 S 
12 14 5 



1 2 3 4 5 
1 2 3 4 5 
12 3 4 5 



1 2 3 4 5 
1 2 3 4 5 



1 2 3 4 S 



1 2 3 4 S 



I 2 3 4 5 
1 2 3 4 5 



1 2 3 4 5 
1 2 3 4;S 
1 2 3 4 5 



1 2 3 4 5 
1 2 3 4 5 



1 2 3 4 5 
1 2 3 4 5 



1 2 3 4 5 
1 2 3 4 5 



2 2 3 4 s 

I 2 3 4 5 



1 2 3 4 S 
1 2 3 4 S 



1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 



1 2 3 4 5 
1 2 3/4 S 



1 ? 3 4 S 
1 2 5 4 5 



1 2 3 4 S 
1 2 3 4 5 
1 2 3 4 5 



1 2 3 4 S 
1 2 3 4 5 



1 2 3 4 S 



1 2 3 4 5 



1 2 3 4 S 



1 2 3 4 5 
1 2 3 -4 5 



1 2 3 4 5 



I 2 3 4 5 
I 2 3*4 5 



1 2 3*4.S 
1 2 3 4 5 



1 2 3 4 5 
4 2 3 4 5 



1 2 3 4 5 
1 2 3 4 S 



1 2 3 4 5 
12 3 4 5 
1 2 3 4 S 
1 2 2 4 5 
1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 
I 2 3 4 5 
1 2 3 4 5 
1 2 3 4.5 
12 3 4 5 
1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 
12 3 4 5 



1 2 3 4 5 

- -i 

1 2 3 4 5 



J 2 3-4 5 
1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 

1 2 3 4 5 
12 3 4 5 

2 2 3 4 5 
I 2 3 4 5 

1 2 3 4 5 . 
1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5:. 
1 2 3 4 5 
I 2 3 4 5 



1 2 3 4 5 
I 2 14 5 
1 2 3 4 5 
1-2 3 4 5 
1 2 3 4 S 
1 2 3 4 5 
1 2 3 4 5 
1 2 1 4 S\ 
1 2 3 4 S 
1 2 3 4 S 
1 2 2 4 5 
1 2 3 4 s' 
1 2 3 4 5 
I 2 3 4 S 
1 2 3 4 5 
1 2 3 4 5^ J 
1 2 3 4 5 
1 2 3 4 S 
1 2 3 4 5. 
1 2 3 4 S 
I 2 3 4 5 
1 2 3 4 5 
1 2 3 4 3 
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Which of the task(s) rated do you Feel require additional training time? 
1. ' 

: — * : - •'• V 

— ■ 1 f m : " 

*• : . . ... 

5. ■ 

6. . . ' ■■ ' * 

.:".->.. - . .. . .. •**'.• . " • ',, 

* List any additional task(s) which you feel need to, be included, in the 
training program. : - _ , 

i. — ■-- — - • ; ■ ■ m , ■ * ■ 

• — ■ — — . ■ ' . ' 

3. i - e ' - " 4 ' .. 5 :' 



5: _ 



Comments : 



ERLC 
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APPENDIX C 



TPE DATA COLLECTION .FORMS USED DURING THE 
EVALUATION OF THE Ml' QT-Ilt' TRANSITION TRAINING 



V 
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oaSERVATtOS OF TRAtKiNG EVENTS - PART I (Incrbduction/Ovcrview/Prerequisites) 

. . XM1 Tng 39 

' Class/Lo^on Title . L—l 1^' 45£ 6: *E 

Training Luca t ion/ EriV iroriinent — — / Dace: - . 

Class Start Timc_ _J J Data Collector ID ff- * 

Class" fenJ'-.TXe o . .Instructor ID // 



OBSERVATION 


YES ' 


NO 


N/A 


ASS 
OK 


SSSMENT 
NOT OK 


* CO.MMENTS ' . 
(Continue Below if 
Necessary) * 


U'crc Students Told 

the Training Objectives? 














Was tlie Purpose of the 
Training Objective provided 
to the students? 








i 


I 


\ t - ''* 

' r ■ 


Kelat iohship between the 
Training -Objective and 
other events was explained? 














Positive or Negative 
.Consequences for ieaxn-ing 
or not learning wer^^ 
provided? 














Was the Qutlixie of class 
activities and schedule 
provided* the students? 














Was terminology which 
will 6*£ r used in the 
instruction explained 
or clarified? 














Questions encouraged 
and answered? 















ADDITIONAL COMMENTS 



FH Form 2951-40 
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* ■ Oli S EiiVAT I QN OK TRAINING EVENTS - .PART II (Demonstration and Isolated Practice) 9 

'■■<-: I \ ' • XM1 Tng 39 

K , . _ 1 .... . ..." " # / ' ■ , ' 

Ciass/tc*?son Ttcic — • ,L^:_^ : 1j Date: - 

• ~— ' : — ■ — 

- • . . .. 5 f 



^ UUS'nKVATljQN 


YES 


NO 


N/-A 


ASSES SMENT- 


COMMENTS 


OK 


NOT OK 


> 1 ' ' 


Were Tasks Demonstrated? 














Was Demonstration conducted 
in small enough steps?' 














Was the isolated practice 
at a high reality level? 


? • 




f 








Isolated practice allowed 
for a range of examples? 


■ *x 


* 










Students were provided 
feedback on their actions? 














Faulty performance was 
identified and corrected? 














Was sufficient repetition 
allowed? 














Were students encouraged to 
ask questions? 










-s 
4 




Student questions were 
answered? 














Were Job Aids introduced as 
Part of the instruction? 














Level of reality progressed 
from tow to High? 















ADDITIONAL COMMENTS 



FH_Fbrm 2951-40 . " - ; Page 2 of 4 

(0T 29 Aug 80} . . 
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; OBSERVATION OF TRAINING EVENTS - PART III (General) 

* • v. " "t - i .,. , XM1 Trig 3§ ; • 

C la s sVi.c s so n : t i c 1 ^ • [ Da t d\ 







N/A 






OB5ERVATT0N - 


YES 


NO 


COMMENTS 


OK 


NOT OK 


Instructor, followed lesson 

plan?. 








9. 




' — 


AV called for was used? 














Training materials and . 
handouts called for were used? 




< 






- ; * 




Instruction was logically 
sequenced? 










- 


- 


Basic rules presented before 
exceptions? 














Learning Objectives smoothly 
transitioned from one to the 
other? t 






• 


9 





i 


Were Critical Discriminations 
emphasized? * . 


> 

— 


-r—< 

- 










Need to Know was emphasized * . 
and nice to know minimized? 














Lesson^ were performance based 
arid' not subject, matter based? 










:■ tj 


1 


j_ ,4_ _ . — : : — * — — 

Students' act ively participated? 














instructor provided a summary? . 














Sessions start/ and q$d on time? 














..t . J* - n ' " ' - . ;■ 

Was instructor attitude 
positive? 















ADDITIONAL COMMENTS 



FH Form 2951-40 ^ Page 3 of 4 

(0T 29 Auq 80) " 44 ' 



- ;■ !j - ' _ XM1 Thg 39 * 

'OBSERVATION OF TRAINING EVENTS - PART til (cotlt) 




OVERALL <RATtN6 GF THE CLASS OR LESSON 



| | - EXCELLENT 
(~) - COOD 
□ --FAIR 
n - POOR 
•' Summarize 



No performance problems predicted 
Few performance problems predicted 
Considerable performance problems predicted 
Widespread performance problems predicted 
r POOR ratings - 



FH For^2951-40 
(OT 29Wuq. 



> 



Page 4 of 4 .. 
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*>usi : ;i<v avion <ir i'io\c;tlci:/TKSi» events 

i:la:;:*/U.e:»soi» Ti LI*- 



Trii i ii i U>ca t ion/ftnvironmcnt 

Test .Start Time 

Test: End Time 



-19L 19K 41>E 63E 



XMl TNG 40 
Pago 1 of 2 * 



Date: 



.1 



Data Collector ID#_ 
Instructor IDSf . 



1 . 
2. 

4. 



Test Has is? 



individual 



Crew 



Field Performance ^ 

NO *k — ' BY WHOM? 



Simulation 



Platoon 



Ik Test Being Scored? YES ■ 

, X * _ _ 

Is Criteria for PASS/FAIL Documented? YES' 



NO 



Have^the Students been* given the criteria? YES 



NO 



'i. Do Guidelines for the Test Administration exist? YES 

, • * s WHERE 

f». is this remedial or POI . Training? 



WHERE 



NO 



OBilKKVATION . 


_YES^ 


NO | UNK 


N/A 


COMMENTS _ 


Did the test occur soon after the 
completion of training? * 












Wore Instructions Clear? 








S 




Are PASS/FAIL Standards Clear? 










p 


Are PASS/FAIL Standards Fair? 












tevel of Reality is as close to 
real world as possible? 












Test sequence v is the same as in 
real world? 












Are critical discriminations and 
responses called for? 












Test calls for integration of 
tasks that will be integrated * 
in the real world? 

■ ; ^ 












Were the specified, tasks testeij? 

■*r - - — 












-f 

Y\\ roKM 295 1- 41 


Is 
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TNG 40 (CONT) 
Paqe 2 of 2 







NO 




fi/A 


'1 

COMMENTS 


OBSKKVATION - 


YKS 


UNK 


Wore the upecifioa :;r7:*ufcn a ci3 
applied? 








4? 




Arc scorers clift errant pojcsoi»!»t::l 
than instructors? 









/■ 




Wu s pc r f o r mo n ce on !;a ttihe i te d ? 
i 












Were students qivrm feedback on 
their performance after * 
Resting? 


i — 








1 



Approximate Number of First-Time Overall NO GO'S 



OBSERVER'S OVERfct*/ itAVSNG OF fHK TEST, 



I) 

u 
u 



GOOD. 
KAIR 
POOR 



No ptrirconoariCC 
Few Performance Problems 
Considerable Performance Problems 
Wides£re#d Performance Probiejas' 



Sanatarizc FAIR or POOR ratings:: 



Ml FORM 2951-41 
(OT 6 ftdg 0?) 
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iUjUtfiNc; KNV,IRQNMENT 
Class/Lobsbn Title.. 




* XM1-TNC-41 
J^9L 19K f 45E 63^ ^ 



Training iJo'ca t i oh ; _ 
Number of Students 



Dace: 



a Number of Instructors/AI 1 s 



Data Col-lector ID#_ 
Instructor ID0 — 



JiWV I KONMk'NTAb FACTORS 



itruc^c 



Studcn,t- Instructor Ratio 



NOT OR 



COMSESyS ' 



Student-Equipment Ratio 



Access to- Instructor -V 



Access to* Equipment 



-Sufficient Training Materials 



Publications Utilized. 



Training Aids 



No. of Students for Spaqe 



Noise Distractions 



Observed Distractions 

' / 



Interrupt ions 



Lighting 



Temperature 



Length of Training Event 



Overall Rating 



48 
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OTHER FORMS , USED DURING THE Ml O^-III 
FOR COLLECTING TRAINING EFFECTIVENESS INFORMATION 
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: DLMOGKAPHIC PROFILE DATA s||E7/QU£StiOSNAti^* : r 

" ^ iON'S-: fiii in front side of _ form r f rp:a.*wj^e4 obtained f-^prn personnel records. 
'■'^ of toirrr. during interview; also* during interview corvfi rri data on front 'side 



Ice-t IT ,Cv.cibti number : 



10. 



EDUCATION 



: Lasc/JiriWSr 
Today Dace:- 



... re Departed ' Unit: ^ 



Rank: 



Classification Battery Scores: \ 0 



5a , fc C0_ 
:c ST 



og . GM_ 

- 6h SC_ 
6i- MM 



"I 



.a GT — 



63 Other- 



iftz: (Enter exact height oa 
• J appropriate line) v 

:j. 5 , O m -5 , 4 m 

7c 5 , 5"-5 f 9 M 

7d 5 , 10 ,, -6 I 2" 
7c;. over 6'2 H 




i^t^ht: renter exact voight oh 
* appropriate line) 



ih ^i/.^. 125 lbs 

:b in-.L30 lbs 

;c 131-^75 lbs 

:- ; fJ&-Z0S lbs 

- Lbs 



■*•■/ Months : 



ERLC 



Y K (BASIC) -• • 

Y cxmi, SGH) 
^Pa y, .Grade -. 1 '• v '- ' 



' i2. ; Permanent MOS 
* (PMOS) : ' 



Day/Mqhth/Year 



J.3 - CURRENT .UgT IT 



(CO) 
(?LT) 



14. Other' MOS (QMOS)- 



i5i BaSsic Active Service Bate* 
; • (Eiiter;'Day, Month, Yearv 

bn appropriate line) 

_i 15 a Before 1260_ 

15b' 1250^1965_ ' : _ 

15c 1956-1,971 __ 

15d 1972-1978 " 



15e After 197S_ 
16. JBirthdaca: 



Day/Month/Year 
■i7i B ? r ?^P^ 2 ? 2: ' _ (Enter state • 

check appropriate Country) ~ 

17a : North" America. _ 

17b South/Central 

America ^ , 

17c Carri'ooean - 

17d . Europe „ ■ , 

17e Pacific ' - ■ : 

17f Other 



18. Physical Profile 



wast type 



19. 



Total Military 
Service: ' 



50 



Years/Sonrhs 

best copy 



2G. ZjZ.Cq :or: 



21. S'^cUricy' Clearance: 



XHl-TKp-1 



"2 la ( .' ) Top Secret 
lit? ^ ) . Secret ' 



z2* \fndt is your present $iuty position? 



v > 



• 22 b 

V 22 d 
^ 22e 



( ) Gunner ^ 

( ) Loader 

( ) Driver : w 

( ) Tank Corraander 

( ) 63 E Mechanic 

( 5 45 E Mechanic "S 

( ) 63K Mechanic' 



23. .Wiac is your Marital Status? 



23 a 
23 b 
23c 
•23d 
23e 



( ) Saver Married j 

( 5 -parried and wife, is-^ith 

( ) ^Jarcied but wife is not with 

( ; \ Divoroed or widowed 

( • ' y LegallV Separated . .;' ' 



21c .( ) 
212 I 5 



22h ( 

22i ( 

2?j < 

22k :.< 



How much longer do you expect. to remain^in the Xray? 



24a 

24b- 

24c 

24d 

24e 



i 



( 



y 

r 

) 



5. or raore years 



:' ) 
) 
) 
) 



221 • C > 
22a t ) 



2 — - 3. yeacs' - 
fe- 2. ^y ears J: 
I. ass than, on^ 



vear 



25. Kbw long have you held your curjrenc pay grade 



25a 
25 b 
25c 
25d 
25s 



c 
( 
( 



3-5 or> more years 
) — 3— 4 .years- 



Confidential 
None. 



63G Mechanic 
45k Mechanic 
34G Mechanic 
Ocher; * 
Explain: - — 
63C ■ 
45W 




. 5 ■*»" .. 



) — 2--- 3~years 

( , ) Lass- tha^w^ vMr . 

v r ' ; - 3' --, ' " - 

How long have you been* in your present duty position (fotal experience, including 



26. 

cthef unit assignments)? 



26a ; ( ) Six months or less 

26b . ( ) 7 - 12 months ._ 

26c ( ) More than 1 but icaa than 2 

26d ( ) More than 2 but less than 3 

26e ( ) 3 or nore years 



Td Forn 2951-1 
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63 



best copy $m ?m 



XMi-TNC-1 



27; How long have you beeh^a member of your, present crew* group or squad? . 

. : : 27a ( ) Less, than ^months S _ • ■ ^ 

27b ( ) 3 " ~ ^ months ' • 

27c ( ) 7—12 months 3 „ „ : 

27d ( ) 13 - IS Months - - 11 

2>7e ( ) More 7 than 18 months 

2S. How long, have you been in your present company* troop or battery? 



ft 



r: 



28a ( - } Les&. than 3 months 

2Sb ( ) 3 - 6 months 

28c ( ) 7 - it months 

28d ( ) 13 - 18, months 

28e ( ) More than 18 months 



29. 



How : long have you tteen in this battalion or squadron? 

29a« N . ( ) Less than 3 months 

29b ( ) 3-6 months 

29c ( ) 7-12 mmths ' <* 

29d • ( r X 13! -. 18 months . t ^ - 

29e ( ) ;More than' 18 months f 



30^ v^What ancteatry do you consider yourself? 





ho*" 


( ' 
< . 


) SpaAish descent 
) American' Indian 




' 30c 


( 


). Asian-American 




~3Gd 


( 


) Puerto &ican 




'.: 30e 


( 


) Philippino 




:' 3QrV 


( 


) Mexican-American 




- k 3 ° s ' 


< 


J ESkimo 


i 


% 30h 


( 


} Aleut 4 . 




* £0i 


( 


) Cuban-American' 




? '< 30j 


( 


) Chinese 




■ 9 36ki 


( 


) Japanese 




/ 301 


( 


) Korean 




30:5 , 


( 


5 Black 




i .; 30n 


V 


) Other, •} 



1 31. : Which hand 
■ . * 3Lb \ ) 



do you use foi doing care'fui work? 

Left hand v 
Right 'hand 
Either hand . 



FK Forts 2951-1 
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i 

32; -What kind of *corx2ctivc lenses do you use Co perform in your duty position?; 

32a ( ) Glasses sometimes 

32b ( ) Classes always " v 

32c ( ) Contact lenses sometimes' 

32d ( ) Contact lenses always 

- 32e ( ) Do not need correction ^_ 

23. How many dependents (not counting yourself) do you have? 

33a ( 7 5 None 

33b ( ) One - f 

33c ( ) Two. ' . . 

' 33d { ) Three . D ■ ; ' * 

33e ( " ) F5ur or mp^e 
*•»■.'■ 
34. What civilian jobs or trades have 'you worked at or had training? 



' 34a ( ) Mechanic 

. 34 d" C " ). Electrician '■ , 

'34c; ( ) Construction * • ' 

,t s 3%& (v 3 ^Iachitle Operator 

\ 34e ( V ) Truclc driver $ *> 

| -35. What ©tfter MOS' formal training do you" have? 

35a , Whdre 



* 



■k 



35b What. Unit ■ 

35c Duty* Position 

35d -How many monphs^ 



What primary MOS formal training do you have? 

36a Where . * ^ ' 

36b What unit. ' 



36c . Duty" Position ' ■ . _ -- l {\ 

^ 36d How many months : ■ ± \- 

37. What doty MOS formal training do you have? 



37a Where - 

37b What unit 

37c Duty Ppsition _ 

37d flow many mqnths 



38. What othet ^OS experience do yoa have? 
38a Inhere j^j> 



58b What unit _ 

23 c Duty PbsfSgJon _ 
38d Kbw many <ndtiths 



FH Form 2951-1 
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■ . - / -- \ 

39. What primary MOS experience do you have? 

... •'• : r - 4 i -* 

39^ Where f * 



39b What unit 

39c Duty Position ; 

39d How many months . . 



40. What duty MOS experience do you have? 

40a Where .... ■ . 



'46b What unit 



40c Duty position 



40d How many months 



41. Are you an advanced individual -training (AIT) or bri^ statioa unit training (OSUT) 
gtatfuate? „ 



41a ( ) AIT 
41b ( ) OSUT 
41c When: 



.1 



inclusive Dates 
41d Where: - ' 



42.. What ARTEP experience do you have? 

42a Where ' ^' * ' . • •: 

42b What unit ■* ' 

- .* 42c Duty . Position ; '_ 

42d How many times ~ . . v 



43., Whar Complete Tank. Gunrxery experience t do you have? 

' 43a Where - . . : > » — - — — -— 

43b What unit . . <. * - • ' 



43c Po,ty- position. _ 1 _ 

43d How many times 

V4. Wha,t < maintenance experience do you have? 



$4^ 



4a Where 



44«b What unit - 
44c Duty position 



44d How many months A 
45. ADDITIONAL. COMMENTS:. " 



DATA COLLECTORS NUMBER ■ • 

JATA COLLECTORS NAME ■ 

\\ _, LAST/ FIRST/MI 

DATA COLLECTORS R^NK -■ 



5K Form 2 ¥5 1-1 % ■* { 

- (07 8 Apr h0) . * - 
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" ; XM1 TNG 37 

V ; TRAINING AIDS DATA SHEET 

* V ' ' • -v ■ * 

Class/Lesson Title 1 19L19K 45E 63E . 

, Class/Lesson H -- ■ — > 

Training Location/Environment Bate: : ,.\ ■ , 

Test Start Time ; Data Collector ID* _ •■ . 

Test End Time . : , Instructor 100 . — 



1. List' training aids required by lesson plan.. 



2.' List training aids used during class. 



> 3. What recommendations do you have for more effective or additional 
training aids? , ' ----- " — ■- - - - 



. 4. Row effective was the instructor's: use of the training a'id(s)? (Check 
one and explain) 

a. () Very effective 




b. ( ) Somewhat effective 
-"' - c. ( 3 Somewhat ineffective 
d. ( ) Very ineffective 



FH Form 2951-37 
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5. Mow effective was the "training aid(s}? (Check one and explain) 
a» ( ) Very effective - 



b. ( ) Somewhat effective 

c. ( 3 Somewhat ineffective x 
d;' ( } Very ineffective 

6. Co'ur.ients: > '. , • ; 



PLATOON CHIEF DATA COLLECTORS FILL OUT FOR EACH- CLASS. LESSON PLAN SHOULD BE 
COMPARED TO POI FOR ACCURACY OF DATA 

- . ■ .. . . ' •" * : - . , i • * ' 
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. i-.i' ' , 4 ^14 PCT 80 

- * ; 14 OCT 80 

INDIVIDUAL TRATNINC/PERSdNNEL STATUS REPORT . . \ ° XM1-TNG-38A 



TITL E- ----- ' ? / ; _ , , : . i 

ACTUAL -TIME (MIN) — — POI TIME (MIN) ^ COLLECTOR ID ff : ' DATE /_ / m 

TNG /TEST/. ' • . . ' TNG TEST • . ■ , \ - ' " . . DD MM YY • 



POS 




rank' 


MOS 


jco 

li" * 

£ — *- 






TIME MISSED ■ 




* 

REASON ABSENT 


• NAME 


ATTENDANCE! 


TEST- SCORE 




, - - .■< 




















— j ■ . 




















— , C 


















































* 










































. 1 








t 




































-* - 














1 -. 

4^ 






















"V. 








t * 




















•■ * - H 






























~* 


t 1 ' i 















































: A=GUNN£R B= LOADER C=*DRIVER .' MECHANICS: , E*63E F=45E G=63H H=63G I=45K .J=34G L=63C M=45N. K=OTHER 
' D-TANK CMDR . < .■ > * MECHANICS: E=63E, F=45E. G=63H H=63G I=45K J-34G L=63C M=45N K-OTUER 

.NUMBER OF RECORDS=00065NK CMDR ' ' " * 
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TANS CREW PERSONNEL STATUS REPORT 



Ci^ss/L^sson Title- 
Class/Lcsboh # ... 
Start Km ; 
End Time 



Dace 




Data Collector ID # 

instructor ID # 

Crew ID $ 



er ID if 



Replacement (Check' if Yes) 
YES ^ 



>v t 3 



Jriyer, ID # ;, s, : /' 



* Reasons for regular jcfw' inanjbers "not Being present for training; 

_ ^ - [ ■ n Hospital-: K ; 

_^ ■ Emergency Leave 

- - >•?' "_ ^ Confinement 

^^^^^^^^^ Other (explain) - > ' ; 



i 1 - 5 

: I 3 



Performance Evaluation Score 



(1-7) 



Number of times exercise performed 



Vll FbKM 2951-39 
(OT 7 Nov 80) 
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' DATE _ 

Please 

3elow, 
today, 
of the 

^ 1 

2 

* 

3 
4 

• 5. 



Sta^slT OPINION QUESTIONNAIRE 
19L- • 19K . - 45E 



Xfii-TNS-42 • 



63E 



.ell us what you think about the class' you -just a'ttended. - Be Honest! 

»•>_" _ • __•____ - - - ' - 

in the space provided, list the titlefsj of the class(es) you were taught 

In the columns provided, write the number that best Shows what you think 
dlass. - • . 



GOOD v 
OKAY- ; • 
BORDERLINE 
BAD - ' 
TERRIBLE - ' 



(No problems - everything, was clear) 

(Few problems ^..almost everything was clear) 

(Rarely acceptable-an average cTa$s) 

(A lot of problems-very* 1 ittle was clear) 

(All probl ems-'pothi ng was clear)- 



•v. 

J, 



p >4 



CLASS 
TITLE 


TRAINING j 
AIDS AND" 
TOOLS 


s " MANUALS ■ 
•• WORKBOOKS 


i 

' PRACTICE 

^ .-TIME 


: TRAINING 
METHOD 


HOtJ.WELL CAN 
YOU DO "THIS 
•TASK? 






' . i *' ' 


' \ 










— — — ' 




~ : t' ■ ', 








— t" — * — 














> - ■ 
*■ ■ * 










j — - — ■ ■ ■■' ■ — 

.. 4 












— ■ — r : — " 


T- _" ' 







: WHY? 



• . r *• - ■ _ 



: 3 \, 



f • # ■ 



V ' s_ 



'4 <T 



; _ + * 



*>2 



- r 



7* 



H FORM 2951-43 
DT 4 AUG 80) 



•7 
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XMI-TNG-M 



CREW STATION TEMPERATURE ASSESSMENT 
DATA COLLECTION FORM 



*4 



DATE: 



PERIOD: DAY (0600-1800), NIGHT (1800-0600) TANK NUMBER 



CREtfoi ID If 



CREW POSITION: C G L D (CIRCLE ONE) DAtK COLLECTOR ID 



MAIN TANK MODES USED DURING THE PERIOD,^ . V - 




<W , E- _ 

'COMFORT JUDGEMENT \ • 
(CHECK 0NE5 ' . «■ : 




HEATER 


BLOWER 


MOBILITY 








OPEN 


CLOSED 


ON., 


OFF 

— i 




FAN 




notv-. 


MOPP 


TIME P 


ERien 






.Comfortable 


hot 


VERY 


ON 


OFF 


MOVING 


MOVING 




— -FROM 


TO 


COLD 


COLD 


HOT 














r i : 






L . ' 


t ■ 
























•r - * 








** 
























"' ■ * 


-r— 




*• 

- * 








- ! 


v s; ■ 


n *rr. 














\-t— r -r 


.1 

k- * 






















H 




a ' 








*. i 






— i 


V 












/ , 






'» 




V'!'. : * 










i "' * 


* "fr 














— H 

























. t' 



, 'to; 



A" - . '■ W- 
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XM1-TNGf*l4 



i ns i. in ir miwjets t udnnaire 



Class/Urssbn 



19L 19K 45E . :63£ .. 



DATE 1 : ~" ; '__ 

INSTRUCTOR ID?? 



Plcii:;c toll us what you think about the class you just taught. Be- Honest! 
1. Did you hawa. instructor '.s material, for each, task you taught? 

.. Yes - ' "'" ' ••- V" * ' 

Did not for these tasks^ — 



2. Do you understand the information in the instructor's materials? 

• >\ 

Yes ' 

. . — ■ 0 

' Didn'.t understand it for these tasks: - — — 







Was enough 'information given -to you? - _;i 

■ / " . Yes * • • 



Neecjed more info for these tasks: 



.4* '■■ 



•~ J r- 



■y^. vj, w r is the information *i n the instructor ' s Ina terials. technically accurate? 

yes 4 • : 4 , ,- v \ 



Inaccurate /nfo for these tasks: 



* A 



/ f-Vv ..... . ... 

. I'll , -mUM..?«)SJ r /J r » 
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f>.* Uitf you have student material for each -task you taught? 
Yes 

- ^ : ■„_,:. " 

- Didn't have material for these tasks; . ■■■ , . 



6. What was the quality of the student material? 



Good 

1 

OK 

Borderline 

Bad ■ 

Terrible » 



I. Did the 'course design "ija^V^faf^^^Sugh practice exercises? 



Yes 



# 



More practice needed- for tftese t tasks : ,_ 



— *r 



. 5 



8. Did the practice exercises - called for by the course design incorporate 
enough realism? 

-Yes V' • . - . 

«*More" realism needed, for these tasks: • 
—?rr>' ~ • • < ' > ; r J 



Yes 



{&f Jl AtJC 80) 





• * V ■ . 










•r..: 
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10. .\.Wihit tusks* if any > did tlie. student scciii to find difficult to Ibarri? 



\Y. 'for the tasks you listed in 10, why ab you think they were difficult to 
earn? 



12; flow would you change this course? 




A- 



-■ ' 

A ■- 



HI KUUM aU'il-flS 



f*. 



.'-1 
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: : i ] - XMl -TNG-5S 

•fjr ; ' . ' Page 1 of .3 




TANK CflEW. FIRINS INTERVIEW 



* • 

Efj.ST TITLE" 



'; CREW ID I ■-. ' ■ • •_ : INSTRUCTOR ID § 

7 



CCLLECT3R ID § 



v • • ' f _' • - ;= 

1 . Did you experience problems with putting the computer into operation? 

•„ ... ; If yes /explain : • '\~ ■ 



'i( ) Yes ( ) No 

y;. '■, ' 



— : — ; ~~- — _ -t— r - 



v ■ v ' • c a 



2. ;Did you experience any problems with the range finder during firing? 
If yes , explain:' ' . , ■: 



( ) Yes ' ( ) /No ; 



T *•( r ^es . ( ) no - * : • . " 





: £ 3. ■'■Ij ere there any pppbl ems with continuous lead? If yes, explain; 

i . ; - ■■ : . ; .^.rr- - • " > ' ' "" • " 

' 1 ' — 7 r ' " : — : f * ■ : : — ! • — 

. ..• .. ' ' - — ---- T- - '- : - : ----- ' 

i ■ t : \ >% re thr $ re any P r 9 bl *ros with thfe sights (TIS„ GP$ f 'GAS)? { explain Vf 
: jC .a^O^^^ ani^irew .action-:* ■ ' i *S : . " \ \ ... ^ V-\ ; 



: — k£ 22 



rr^ • ' , jfr* — — — : — : "~ 
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VCiMt actions did. the crew take with- prob^mst* (ReQues ;3L-4J 



Corrected by Crew Reported to Ma£nr ■ Fired Anyway*. 



_ _ _ _ ... ._•».. 

V>. i-.'a& thdro .an indication of a fire control system failure displayed. ■ j ; • * * = - 

tSiJ si>-ht during firing? If yes, state what failed and what action was >•..- * 

taken by the crew: «, -' ' 6*; ^ - ; _ 

( ) Yes ( ) No • . ' * * ^ 1 v 



7. Was th^groper ammo indexed for each firing 
( ) YfiSr^ ( ) No ( ) 



■Y i. 



-V - * 8, Did the tank commander fire Any main gun *ou»ds *fr"o» Jiis oVerr±de£.. If 
yes, why: ^ . j .: V=:.' ■ C ^ V < / 



( ) Yes ( ) . No 





** 








** » 









^ * '9. D$d continuous lead make it easifer for theAgunner to hit thW 4 „ 

( . ) Yes * ( So,, \ , .i. "* . " - i,, 

■ --„- ' 'A* -• i * i - * etf ■■ * ' ^ ' " ■ 

10. Was emergency power switch used during the firing? - i , • i - ,* v 

(9 ;Yes ( ) No If yes, why? ' ^ vl ' / >' •■ ' 



w - >• v ». ■. - r _ • • \ : ■ . . . ^ ^v- - • \ 

11. . Were", there 'any' lifain gan malfatictions during 4 the coarse? ij # f '^eS j. Vhac 
was the malfunction an4 what corrective action was «aken?- • \ '-Vj 



A 



• V- . ~ —7 — ~ 1 — " • r— — ~jp T. 1 " 

i 3 "FH FORM 2951-58. 1 ., - ? " .• '• / a • , • 4 ■ 

(CT 29 Aug 80? '* ' i ? .j* _ t 




• *12 During night operations, did ,che driver. use hi& night. 'viiion viewer? 

( .) Yes ( ) No . C - . ^ , '"■ 

, . . . \ ■ ju ... ... , . ... . 

J3. Merc there any problems installing "'the driver's night vision viewer? 
( if yea* 'wha^ the .problem? 



I ' Yes 



< 3 No 



i' : — 



: > 



t4. Wa«?<h4 driver's hatch closed>during firing? 



(.:.)■ Yes. 



( 5 No 



^5.;. Wftat^as tfce crew's impression of the computer? 



T 



'. ■ ;s *• •• 



* TanV 



r-- 



- 1 ■ \* 



^ J Itf; ;_. Wh^t V is^ e^cH crewman's overall iSression of the XM1 fire control 

system?;- 



DriA/eiv 



'I L4ader / 




" Superior' 


Good 


Fair 


Poor 


Bad ' ; 


• ... 












.' ..5 






' . ( ) 


( ) 


( 


) 




> 


.( .3- 




f' ; i •) 


( 3 • 


i 




f 


3 


. < 3 






















( 3 


( 


,3 


' & 


3 


( 3 




..< >"* 


< ) 


( 


1 


( 


3 


( ) 





■tr 



/ 
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: : — . \' APPENDIX E * ' , •'• ■■• • 

• - •-. - • ,, \ ■ . ( 

' 0 ~ ' ■ - LESSON PLAN EVALUATION CRITERIA ■ ' J . ' . * 

' " ■ ' >.-: -. v . ' . - ' . 

" - ' - • • ■ u . ' i> : 

The training' objective should specify x^ha t :the is taden t mast do after having 
been trained'. ^ ' :■ ' * l 

2. • Training objectives must not be/confused With jab requirements. Training 
btijebtiyes specify" behaviors that stu(ients„are expected to , exhibit after train- 
ing. These behaviors are r*;. pe^sar ily tfte. sama as the Sehaviots that are 
required on the job. * 5^ : '^i ^ 

** 3. A training objective shauliJ^spec iJfy the conditions, under which "the student 
.must demonstrate .tjas^k; performance. '. ■-. . f . ; ■> 

4. Condition*** seated 'as *part A of # the training \oblfcctive must be, clearly train- 
ing related fratherV than' 3©b related}..-. '■ s • . ' ' 

~ i. ' , • <* • • . •' bt > s . 4J U :„. ' • . 

; - V ■,■/-■-;*; v^T M " \ * \ > - : -.\ , v 

5. '^The^ trc/ind'ng^ 6bgec tiVe shotild ^ecif^, .the standards to which the studenit ; 
must perform*. * [The • s tandards- shduld :fce - clearly s pel le$ out so that the student, 

<^the iris'triic to*x% and' the traini f tfg^ valuator cot 'tfell^he- difference between ■ . 
performance at or ibo\rc standard fx^cfci performance that is belpw standard. ' : \ 

. --' • , J. - .^>?y .- * ■- . 1 -j_ • -j. ■ > _^.„ • -v. - . ). 

1 The -test items ^puld derive djlredtly f ram^.ttie training" objectives . " ► 

7. Generally the/ test should^ require t*.he : stude'nt to perform, the steps speci- y • ' 
fied in the training, objective. ' ^>V- . , ' x ? 

. * -r . y^- ^ ^£ : / ... . -'. 

8. ^ The tondit ions^^pe.cif AecUf or the test : ;,shouW„ be the same ' thas'e specified 
for the training., in .the* trashing f objectiv^s. ^' ; ■ . ; .* v 

c .' ' . _ ; ■ ■ ' * ° : . *i .. ■_ # ' - _• ? 

The standards specified for the test sl^ou Jd* 1 be?" the game as those specified ' /i 
rh the training obje'cti've. ' >*^f ■-- ' *• ^ * " ^ .. * •! 5 * t 

10. . The # instruct ions for administering the test shoul^ be written so as to ^ t ' 
ensure^ standardizatiarn : .,or test adminis'tj^t.ion procedures across instruqtors . - " 
The lesson - plan should outline' prG^i-s^^ (lpv.'the test is to be kdminis tered . " 
_ Included should be th^piacement of personnel • aji-cj equipment and test admin is- " 
5 trat*on procedu res that spec^iiy the te*st^.ng sfeqtiQnte ajrid guide tl^e evaltiator * . -■* - «■ 

in testing .the students on- each of. the * tasks to J?e performed. ^ *' ,V 

...... " . ../ ; - ' -V - - - ' 9 % 

f 11*^ Instructions to th^? students ^oncerrtin^ how the^ testVw^ll, J>e condticted - : "i 

apfl, scored shcy5ld v be included' in the' lesson plali . These" ^r^trufeidns fshouida • . ■> 
Akc tflgar arvd complete' Co ensure tha-t test'ers ^o not have" to acid ins^tructi&hs : 

^of tneir own. ^ ^ \ \ • ' ' % ./ . ' " ; • J- 

■ • ■ -rt v ■ • ■ » 

■ ■ 4 . : " - ■ * ' - ; '-■ % * 

12, The lesson plan should c|H for the d ifj^emin»it ion of enabling 'knowledge ' ^ y M ' 
ancl specify, what this enabling knowledge consists of. (Terminology or. other* * 
|Su6j ect matter tha v t is designed to enhanc^ students 1 * understanding of .the task p * 
i,/to he learned lf> referred to as onabling knowledge.) ' ■ . 

'.* • *"•;<: ' • • • , '• - • ' ■ ■ ■ 

• . ' \- ■ 
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13. .THie. lesson plan sfioiild call for the demonstration of the task (in its 
entirety) . * ' * 

^14. When there are subtasks in a lesson, demonstrations should be required for 
each subtask^ arid an integrated demonstration should be required following 
subtask training. ' * ■ . : s 

15. The lesson plans "should describe how to conduct the demonstrations. 

16. Practice activities should "be" called for in the lesson plan. 
_ * 



17. The lesson pi art should specify that each suStask and task be practiced.' 

18. The lesson plan should provide guidance to instructors that tells them 
what to look for during practice and how to correct faulty performance when it 
occurs.. This guidance might include some ( or all of the following: (1) speci- 
fication of aspects of the tasks tha-t. jnight be expected to cause problems for 
students, (2) common student errors on the task being practiced, (3) telling 
the instructor what to do when the student is unable to even start doing the _ 
task (e.g., demonstrate the task again), (4) telling the instructor to" provide 
additional assistance when student . progress toward the objective stops, and 
£5) directions to mafee^' feedback regarding student errors immediate^ Specific / t 
to the actions performed, and free from harshness qr ridicule-. ' "W.-- 

19^. The lesson plan should specify that practice on each task/subtask be per- 
formed by each student to a specif ied standard (or to the standard specified 
in the training objective) . ' 

' v,V' 
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